Inference API

The inference plane is the customer-facing data plane. It listens on RELAY_PORT (default 8080) and speaks OpenAI- and Anthropic-shape wire protocols, plus Relay’s own provider-neutral canonical shape.

Authentication

Every inference request authenticates with a relay key as a bearer token:

Authorization: Bearer <relay-key>

Relay keys are minted in the admin UI or via the control plane (POST /api/relay-keys). The plaintext is shown exactly once on creation — Relay stores only sha256(plaintext).

Namespacing

Each vendor wire shape is served under its own path prefix. The bare /v1 namespace belongs to Relay’s canonical shape.

Prefix	Wire shape
`/openai/...`	OpenAI Chat Completions, Responses, Embeddings
`/anthropic/...`	Anthropic Messages
`/v1/...`	Relay canonical (provider-neutral)

Endpoints

Method	Path	Shape
`POST`	`/openai/v1/chat/completions`	OpenAI Chat Completions
`POST`	`/openai/v1/responses`	OpenAI Responses API
`POST`	`/openai/v1/embeddings`	OpenAI Embeddings (byte-pass)
`POST`	`/anthropic/v1/messages`	Anthropic Messages
`POST`	`/v1/generate`	Relay canonical request/response
`GET`	`/v1/models`	Models accessible to the relay key
`GET`	`/healthz`	Liveness + Postgres ping (public)
`GET`	`/openapi.json`	Generated typed OpenAPI spec

Examples

curl http://localhost:8080/openai/v1/chat/completions \
  -H "Authorization: Bearer <relay-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "hello"}]
  }'

curl http://localhost:8080/anthropic/v1/messages \
  -H "Authorization: Bearer <relay-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "hello"}]
  }'

curl http://localhost:8080/openai/v1/chat/completions \
  -H "Authorization: Bearer <relay-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "stream": true,
    "messages": [{"role": "user", "content": "hello"}]
  }'

Responses stream byte-for-byte from the upstream when the inbound shape matches the upstream shape. Cross-shape requests (e.g. OpenAI in, Anthropic upstream) are translated per chunk through Relay’s canonical protocol.

System messages

Start-of-conversation system instructions map to the upstream’s native form: the top-level system field (Anthropic), systemInstruction (Gemini), or a leading system message (OpenAI-compatible). System messages that appear mid-conversation keep their position — they are never merged into the start-of-conversation system prompt, so the upstream’s prompt-cache prefix is not invalidated by a late instruction. Relay’s canonical semantics follow the OpenAI shape here: on OpenAI-compatible upstreams the message is sent as a real system turn in place. Anthropic and Gemini upstreams cannot express that on every model, so there the message is delivered in place as a user turn wholly wrapped in <system> markers:

{"role": "user", "content": "<system>\nFrom now on, answer in French.\n</system>"}

The marker round-trips: a user turn that is exactly this form is restored to a system message when Relay parses the conversation back. A <system> tag embedded inside longer user text is never promoted. Inbound Anthropic requests may also carry role: "system" messages (supported by recent Claude models); Relay accepts them and keeps them positional. A canonical system message can instead opt into full system authority with "hoist": true: at serialization it merges into the upstream’s start-of-conversation system form on every shape. Hoisting changes the very beginning of the prompt, so it invalidates the upstream’s cached prefix for that conversation — use it for instructions that must override user content, not for routine mid-session notes.

Models

The model field is resolved against your catalog. A model is reachable only if a policy grants it to your relay key and the model has an enabled host binding with a healthy host key. List what your key can reach:

curl http://localhost:8080/v1/models \
  -H "Authorization: Bearer <relay-key>"

Errors

Status	Meaning
`401`	Missing or invalid relay key.
`403`	Relay key’s policy does not grant the requested model.
`404`	Unknown model, or no enabled binding for it.
`429`	Rate limit reached (relay-side or upstream).
`502` / `503`	No healthy key in the pool, or upstream unreachable. See Troubleshooting.

Relay does not fail over mid-stream. Failover across keys and hosts happens before the first byte reaches you. Once bytes flow, an upstream error is returned as-is.

Get Started

Concepts

Reference

Authentication

Namespacing

Endpoints

Examples

System messages

Models

Errors

​Authentication

​Namespacing

​Endpoints

​Examples

​System messages

​Models

​Errors

Authentication

Namespacing

Endpoints

Examples

System messages

Models

Errors