Skip to main content
Relay is a self-hosted gateway that puts one endpoint in front of your LLM providers. Your app sends OpenAI-shaped (Chat Completions, Responses, Embeddings) or Anthropic-shaped (Messages) requests, or Relay’s own canonical shape at /v1/generate, with a relay key. Relay checks what that key may use, picks an upstream credential from a pool, enforces rate limits, forwards the call, and records usage and cost.

​
Why it exists

A provider API key has one rate limit and fails on its own. Running real traffic means writing the same code in every app: spread load over several keys, retry on 429, skip a revoked key, keep per-model limits, attribute spend to a team. Relay moves that into one service you run next to your apps.

Key pooling

Several host keys serve one relay key. A failing key is skipped by its circuit breaker before the first byte reaches your app.

Cross-shape routing

Keep your existing SDK and change the base URL. A request in one shape can be served by an upstream that speaks another.

Policies and rate limits

A policy decides which models a caller reaches, which host keys it spends, and which limits apply.

Admin UI and control API

Hosts, keys, policies and tenancy are rows you change at runtime. Changes reach every pod in about a second.

​
The mental model

The catalog describes what can be called. Credentials and policies describe who may call it and with what.
An upstream endpoint Relay sends requests to: a provider’s API (openai, anthropic, google) or your own server, such as a self-hosted Ollama. A host has a base URL. See Catalog.
A catalog entry, owned by the provider that makes it. The model field of a request resolves against the catalog by name, optionally pinned to a host as model@host.
The join of one model and one host. A binding carries the wire shape Relay speaks to that host for that model (openai, openai_responses, anthropic or gemini), the upstream model name, and the pricing. A model is reachable only through an enabled binding, and one model can have bindings on several hosts.
Your provider credentials, stored by Relay as an environment-variable reference or as a value encrypted under the master key. Each host key belongs to one host. See Authentication.
The credentials your apps send. A relay key (sk-wr-…) is long-lived and belongs to a service account or a user. An inference token is short-lived, minted for a signed-in user, and scoped to one project. Neither exposes a host key. See Authentication.
A policy grants models, lists the host keys requests may spend, and names a rate limit. A relay key can point at a policy directly; otherwise a service account’s policy or a project’s policy bindings decide. See Policies.
Named rule sets (requests, tokens, concurrency per window). A policy’s rate limit caps everyone it governs; a host key’s tier limit caps that one upstream credential. See Rate limits.
The tenancy layer. Teams contain projects; projects contain service accounts; users join through groups and role bindings. Keys, policies and usage are attributed to this hierarchy, and RBAC scopes on it. See Authentication.

​
A request, end to end

1

Authenticate

The app sends a request with a relay key or an inference token. Relay resolves it to a principal: a service account or a user, plus its project and team.
2

Resolve the policy

Relay finds the governing policy: the key’s own, else the service account’s, else the first matching policy binding in the project.
3

Route

The model is resolved against the catalog. Relay walks the model’s enabled bindings, keeps those the policy grants, and picks the first whose host has a usable host key in the policy.
4

Reserve and pick a key

The policy’s rate limit is reserved. A healthy host key is picked from the policy’s keys for that host, and its tier limit is reserved. If the key fails before any response bytes are sent, Relay tries the next key.
5

Forward and settle

The request goes upstream, translated if the binding speaks a different shape, and the response streams back in the shape you sent. When the response ends, Relay commits the token counts to the rate limits and records the usage event.

​
Where to go next

Quickstart

Run Relay and send a first request.

Architecture

The two listeners, the in-memory snapshot, and where state lives.

Inference API

Endpoints, wire shapes, streaming, and error codes.

Control API

Manage hosts, keys, policies, and tenancy.