Troubleshooting

​

401: relay key or token rejected

• Send the relay key as Authorization: Bearer <relay-key> (or X-WR-API-Key, or x-api-key).
• Use the full value returned when the key was created, not its id or prefix. It is shown once; if you lost it, create or rotate a key.
• api key rotated means the key was rotated and the old value’s grace period has ended. Use the new value.
• token_revoked means an inference token was revoked, by id (DELETE /api/auth/token/{jti}) or for all of a user’s tokens (POST /api/auth/token/revoke-all). Mint a new one; see Inference tokens.

​

403: authenticated, but not allowed

​

404 model_not_found

curl http://localhost:8080/v1/models \
  -H "Authorization: Bearer <relay-key>"

​

503: nothing to route to, or relay is overloaded

​

502: every key fails or is cooling down

1. Fix the cause: correct or rotate the upstream key, or check the host’s baseURL for upstream_unreachable.
2. Read a key’s breaker state with GET /api/host-keys/by-id/{id}/health (state, reason, cooldown_remaining_seconds).
3. Wait out the cooldown, or rotate the key’s value, which gets a fresh breaker. With RELAY_STATE_BACKEND=memory, restarting relay clears every breaker. With Valkey, breaker records are the keys matching secret_health:*; deleting them clears them:
   valkey-cli --scan --pattern 'secret_health:*' | xargs -r valkey-cli del
   

​

429: slow down

​

413: request too large

​

408: request timeout

​

Wrong wire format from the upstream

​

The admin UI won’t sign you in

• Standalone image: the admin password is printed to the container log on first boot (docker logs <container>) and saved on the data volume. On a volume that already had a config directory, no new password is generated; read it back as in Retrieving or pinning the master key.
• Compose stack: sign in as admin with RELAY_ADMIN_PASSWORD from .env.
• Lean image: there is no user until you create one. Use RELAY_ADMIN_TOKEN as a bearer token, or mount a kind: User file into RELAY_CONFIG_DIR; see Docker.
• Over plain HTTP, set RELAY_COOKIE_SECURE=false; otherwise the session cookie is marked Secure and the browser drops it. The standalone image already does.
• The admin UI is on the control port (RELAY_CONTROL_PORT, 8081 by default), not the inference port.

​

Credentials and catalog regenerate on every run

​

Inference tokens can’t be minted

​

Stored host key won’t decrypt after a restart

​

The original master key is lost

1. Run relay secret status with only the new key configured and note undecryptableRows. A value’s id is the id of the host key it belongs to.
2. While any value in the format from before data keys cannot be decrypted, relay secret rewrap is refused, so delete the values in undecryptableRows by hand:
   BEGIN;
   SELECT id, name FROM secrets WHERE id IN ('<id>', '<id>');  -- the host keys to re-enter
   DELETE FROM secret_values WHERE id IN ('<id>', '<id>');
   COMMIT;
   
   An id that starts with payload-logging: holds payload-logging credentials: write them again in the payload-logging settings section. Any other id with no host key is the inference-token signing key: clear signingKey in the auth:tokens settings section so the next boot generates a new one (tokens signed by the old key stop verifying).
3. Run relay secret rewrap --to <new kek id>, where the new KEK id is the one in heldKeks. It lists every data key under the lost key as skipped, and new data keys are wrapped under the new key from then on.
4. Run relay secret rotate --all, so every scope gets a primary data key the new key opens.
5. Re-enter the values of the host keys from step 2 and of any other host key that loads unresolved.

​

Wrong master key at the upgrade to data keys

1. Set RELAY_MASTER_KEY to the correct key and RELAY_MASTER_KEY_EXTRA to the wrong one, and roll every pod. The pods convert the old-format values within seconds; wait until legacyRows is 0. New data keys are still wrapped under the wrong key until the next step.
2. Run relay secret rewrap --to <kek id of the correct key> (the first entry in heldKeks). Every data key is wrapped under the correct key and it becomes the target. Skip this step if targetKek is empty or already the correct key.
3. When relay secret status shows every data key under the correct KEK, remove RELAY_MASTER_KEY_EXTRA and roll every pod.
4. If the wrong key may be known elsewhere (it belongs to another deployment, or came from an example), whoever has it and a copy of the database from that period can unwrap the data keys it wrapped. Rotate every scope (relay secret rotate --all), wait until the old data keys show 0 rows, and destroy them, as in If the master key leaks.

​

Health checks

# Readiness: 200 once Postgres answers and the catalog has loaded, else 503; Postgres stops counting after that (public)
curl http://localhost:8080/healthz

# Liveness: 200 while the process serves (public)
curl http://localhost:8080/livez

# Build version (public)
curl http://localhost:8081/api/version