Skip to main content
A provider API key has one rate limit and fails on its own. Relay lets a policy list several host keys for the same host, picks one per request, and takes a failing key out of rotation with a per-key circuit breaker, so one throttled or revoked key does not stop your traffic.
Key pooling and failoverKey pooling and failover

One relay key draws from a pool of host keys; unhealthy keys are skipped automatically.

​
The pool

A host key is one upstream credential, and it belongs to one host. The pool for a request is the set of host keys the governing policy lists in hostKeyIds that belong to the host routing chose, minus disabled keys and keys whose tier policy does not serve the requested model. Your callers never see which key served them. Two consequences:
  • A policy has to list host keys. A host key that no policy lists is never used by policy-governed traffic.
  • Policies that list the same host key share it, along with its circuit breaker and its tier limit.
A host marked noAuth (a self-hosted Ollama, for example) needs no host key: Relay uses a built-in anonymous credential that sends no auth header and is never taken out by a breaker. Requests with no governing policy draw from the host’s system-owned keys plus the caller’s own. Only a personal key with no project can make them, only when the inference setting allowMissingPolicy is on, and never under RELAY_AUTHZ=rbac. See Policies.

​
Picking a key

The policy’s keySelection sets the algorithm: Breaker state is shared across pods through the state backend (Valkey when RELAY_STATE_BACKEND=redis). Every change to a key’s breaker is one atomic script, so pods recording failures on the same key add to one count. After picking a key, Relay reserves the key’s tier rate limit if its tier policy has one. A key over its tier limit is skipped without touching its breaker, and the next key is tried. Each pod then skips that key for the requested model until the limit’s retry time. See Rate limits.

​
Circuit breakers

Each key has its own breaker, keyed by a hash of the key’s value. The upstream response decides what happens: A single 5xx does not take a key out: providers return a few percent of them on healthy keys, and the request already moves to the next key. RELAY_KEY_BREAKER_THRESHOLD and RELAY_KEY_BREAKER_WINDOW_S set the count and the window. Failures of requests that were already in flight when the key opened do not move it further up the ladder. An open key is skipped during selection. When its cooldown ends, one request probes it while the others keep skipping it; if the probe’s outcome has not come back within 10 seconds (a long stream, say), the next request probes too. A successful probe closes the breaker one backoff step down, and each further success takes another step off; a failed probe moves it one step up. The step counter is shared by all failure classes. No failure takes a key out for good, since the request that caused a 401 or 403 may have been the caller’s doing.
Because breakers are keyed by value hash, rotating a key to a new value gives it a fresh, closed breaker. The old record expires on its own.
When the host cannot be reached at all, the key is not blamed: Relay re-dials the same host up to three times with a short backoff, then returns 502 upstream_unreachable. Every key on that host shares the same base URL, so switching keys would not help.

​
Failover happens before the first byte

A request tries at most three host keys. If the upstream answers with a retryable failure (401, 403, 429, 5xx, or a network error) before any response bytes reach the caller, Relay records the failure and moves to the next healthy key in the pool. Once bytes start flowing back, the request is committed and there is no mid-stream failover. Failover stays within the chosen host. Routing picks one binding per request, and Relay does not switch to another host or provider when that host’s keys fail. When the attempts run out, the caller gets the last upstream response with its status and body; when no healthy key is left to try, the caller gets a 502 with no healthy keys in pool.

​
Recovery after a 401

A 401 can mean the value Relay holds is stale, for example after a stored key was rotated on another pod. Relay re-reads the key’s value from its source (the encrypted value in Postgres, or the environment variable) before giving up:
  • Other healthy keys remain: the request fails over at once, and the re-read runs in the background so the key rejoins the pool if its value changed.
  • It was the last candidate: the request waits for one re-read and retries the same key if the value changed.
  • The value is unchanged: the key is treated as revoked, the caller gets the upstream error, and the key is probed again after each cooldown.

​
Inspecting and recovering keys

Read a key’s breaker state:
The response gives state (closed, open, half_open, or unknown when the key’s value cannot be resolved), reason (upstream_auth_failed, upstream_forbidden, upstream_rate_limited, upstream_server_error, upstream_network_error), open_until and cooldown_remaining_seconds. The relay_provider_keys_down_total metric counts breaker openings by reason. If every key in the pool is open, requests fail with 502 and the message no healthy keys in pool. That almost always points at a real upstream problem: a revoked credential, an exhausted quota, an outage. To recover:
  1. Fix the cause, for example by giving the key a new value or adding another key to the policy. A stored key is rotated with POST /api/host-keys/by-id/<id>/rotate and body {"value": "<new value>"}; an env-ref key takes the new value from its environment variable after a restart. Either way the new value gets a closed breaker.
  2. Otherwise wait out the cooldown: at most 60 s for 403, 5xx and network failures, at most 1 h for 401.
  3. To clear all breaker state immediately: with RELAY_STATE_BACKEND=memory, restart the pods; with Valkey, delete the secret_health:* keys.
See Troubleshooting for related errors.