Relay's two planes share one in-memory catalog snapshot, kept in sync from Postgres via NOTIFY/LISTEN.
The two planes
Inference plane
Listens on
RELAY_PORT. Serves the shape-prefixed routes (/openai/v1/*, /anthropic/v1/*), the canonical /v1/*, the client-profile prefixes, and /healthz. Authenticates relay keys and inference tokens. Dispatch is the same code for every vendor; the wire shape comes from the host binding.Control plane
Listens on
RELAY_CONTROL_PORT. Serves the admin UI and the control API under /api (sign-in, CRUD for every kind, settings, usage, apply/export) plus /metrics. Authenticates with a session cookie or the admin bearer token.RELAY_CONTROL_PORT=off disables the control listener on a pod.
The snapshot
Every pod holds an immutable in-memory snapshot of the catalog, credentials, policies and tenancy rows. Routing, authentication and policy resolution read only the snapshot.- A control-plane write lands in Postgres.
- The write fires a
NOTIFYon a Postgres channel. - Each pod’s listener receives it, applies the change to a copy of the snapshot, and swaps the copy in. Notifications are debounced over one second.
POST /api/reload rebuilds the snapshot from Postgres in full if you need it.
A control-plane write propagates to every pod's snapshot via Postgres NOTIFY/LISTEN.
A request, end to end
The inference request lifecycle: auth, route, draw a key, forward, stream, settle.
1
Authenticate
A relay key is hashed and looked up in the snapshot; an inference token is verified against the signing key. A miss, a disabled or expired key, or a bad token is a
401.2
Resolve the policy
The governing policy comes from the key, the service account, or the project’s policy bindings. See Policies.
3
Route
The
model resolves to a catalog model. Relay walks its enabled host bindings and picks the first one the policy grants whose host has a usable host key in the policy. The binding supplies the wire adapter (openai, openai_responses, anthropic or gemini) and the upstream model name.4
Reserve and pick a key
The policy’s rate limit is reserved in one script call to the state backend. A host key is then picked from the policy’s keys for the chosen host, skipping keys whose circuit breaker is open, and the key’s tier limit is reserved. See Key pooling and Rate limits.
5
Forward and stream
Relay calls the upstream and streams the response back. When the inbound and upstream shapes match, bytes pass through unchanged; otherwise each request and stream event is translated through Relay’s canonical protocol.
6
Settle
When the caller finishes reading the response, a background goroutine commits the observed token counts to the rate limits, records the key’s success, and hands the request to the usage, payload and metrics observers. This work never delays the response.
Failover happens only before the first response byte, and only across host keys of the chosen host. Relay does not switch to another host at request time, and once bytes flow an upstream error reaches the caller as-is. See Key pooling.
Design choices on the request path
- No Postgres queries. Routing, authentication and policy resolution read the snapshot.
- A small, fixed number of state-backend calls. The inbound rate-limit reservation and a token’s revocation check share one script call; key selection reads each candidate key’s breaker record; the key’s tier reservation is one more script call; commits after the response are batched.
- Off-path emits. Usage events, captured payloads and metrics go through bounded queues that drop on overflow and count the drops, so a slow sink cannot stall a request.
- Byte-pass when shapes match. Same-shape requests are copied, not parsed and re-serialized.
- Horizontal scale. Shared state lives in Postgres and the state backend, so pods are interchangeable; add pods to add capacity.
relay_overhead_seconds histogram, which excludes the upstream call. See Metrics.