Skip to main content
Relay is a self-hosted gateway that puts one endpoint in front of your LLM providers. Your app sends OpenAI-shaped (Chat Completions, Responses, Embeddings) or Anthropic-shaped (Messages) requests, or Relay’s own canonical shape at /v1/generate, with a relay key. Relay checks what that key may use, picks an upstream credential from a pool, enforces rate limits, forwards the call, and records usage and cost.

​
Why it exists

A provider API key has one rate limit and fails on its own. Running real traffic means writing the same code in every app: spread load over several keys, retry on 429, skip a revoked key, keep per-model limits, attribute spend to a team. Relay moves that into one service you run next to your apps.

Key pooling

Several host keys serve one relay key. A failing key is skipped by its circuit breaker before the first byte reaches your app.

Cross-shape routing

Keep your existing SDK and change the base URL. A request in one shape can be served by an upstream that speaks another.

Policies and rate limits

Policies on teams, projects, identities and provider accounts decide which models a caller reaches, which keys it spends, and which limits and budgets apply.

Admin UI and control API

Hosts, keys, policies and tenancy are rows you change at runtime. Changes reach every pod in about a second.

​
The mental model

The catalog describes what can be called. Credentials and policies describe who may call it and with what.
An upstream endpoint Relay sends requests to: a provider’s API (openai, anthropic, google) or your own server, such as a self-hosted Ollama. A host has a base URL. See Catalog.
A catalog entry, owned by the provider that makes it. The model field of a request resolves against the catalog by name, optionally pinned to a host as model@host.
The join of one model and one host. A binding carries the wire shape Relay speaks to that host for that model (openai, openai_responses, anthropic or gemini), the upstream model name, and the pricing. A model is reachable only through an enabled binding, and one model can have bindings on several hosts.
Your provider credentials, stored by Relay as an environment-variable reference or as a value encrypted under the master key. Each host key belongs to one host. See Authentication.
The credentials your apps send. A relay key (sk-wr-…) is long-lived and belongs to a service account or a user. An inference token is short-lived, minted for a signed-in user, and scoped to one project. Neither exposes a host key. See Authentication.
A policy names the models requests may call, the pools of host keys they draw from, and the rate limits and budgets they must fit. A policy binding attaches it to accounts: a team’s, a project’s, a member’s, a service account’s or a provider organization’s. A request runs in one account and passes every account above it, and each can only narrow it. See Accounts and lanes and Policies.
Rate limits are named rule sets (requests, tokens, concurrency per window); budgets cap spend in dollars per day, week or month. On a caller’s policy they cap that caller; on a provider account’s tier policy they cap every key of the account. See Rate limits and Budgets.
The tenancy layer. Teams contain projects; projects contain service accounts; users join through groups and role bindings. Keys, policies and usage are attributed to this hierarchy, and RBAC scopes on it. See Authentication.

​
A request, end to end

1

Authenticate

The app sends a request with a relay key or an inference token. Relay resolves it to a principal, a service account or a user, and to the account the request runs in: the credential’s default, or one the request names.
2

Read the policies

Relay reads the account’s plan: the policies bound to it and to its project’s and team’s accounts, folded ahead of time.
3

Route

The model is resolved against the catalog. Relay keeps the caller’s pools, in order, that are at a host serving the model and whose policies allow it.
4

Reserve and pick a key

The caller’s rate limits and budgets are reserved. A healthy host key is picked from the first pool, and the limits of its account, its host and the key itself are reserved. If the key fails before any response bytes are sent, Relay tries the next key, then the next pool.
5

Forward and settle

The request goes upstream, translated if the binding speaks a different shape, and the response streams back in the shape you sent. When the response ends, Relay commits the token counts to the rate limits and records the usage event.

​
Where to go next

Quickstart

Run Relay and send a first request.

Architecture

The two listeners, the in-memory snapshot, and where state lives.

Inference API

Endpoints, wire shapes, streaming, and error codes.

Control API

Manage hosts, keys, policies, and tenancy.