Metrics

Relay exposes Prometheus metrics for traffic, its own added latency, load shedding, background record loss and key health. Every Relay metric is in the relay namespace; the standard Go runtime (go_*) and process (process_*) collectors are registered too.

Scrape endpoint

Metrics are served at /metrics on the control listener (RELAY_CONTROL_PORT, default 8081), at the listener root, not under /api. Each pod reports its own counters, so scrape every pod.

curl http://localhost:8081/metrics

Prometheus scrape config

scrape_configs:
  - job_name: relay
    static_configs:
      - targets: ["relay:8081"]

/metrics needs no credentials. Keep the control port off the public network, or put the scrape path behind your own access control. On Kubernetes, the Helm chart can create the scrape config (serviceMonitor.enabled for the Prometheus Operator, vmPodScrape.enabled for the VictoriaMetrics operator) and ship two Grafana dashboards as ConfigMaps labelled grafana_dashboard (grafanaDashboards.enabled): Relay — Service, built on these metrics (traffic, latency, overhead, admission, in-flight requests, queue depth, records lost, keys down, per-pod runtime), and Relay — Usage (ClickHouse), which reads the ClickHouse usage store through the Grafana ClickHouse data source. See Kubernetes.

Measuring Relay overhead

relay_overhead_seconds is the time Relay adds to a request: total time minus the upstream leg, where the upstream leg runs from handing the request to the upstream until the upstream finished sending its response. What remains is routing, rate-limit reservation, key selection, request and response translation, and any failed upstream attempts before the one that succeeded. The clock starts after the credential check and the request body read, so neither is counted. It is observed only for requests whose upstream response was read to the end; a request refused before reaching an upstream, or cut off mid-response, adds no sample. relay_admission_seconds is the part before the handoff. Read it per source, as a quantile over the histogram:

histogram_quantile(0.99, sum by (le, source) (rate(relay_overhead_seconds_bucket[5m])))

Buckets run from 100 µs to 500 ms. The value depends on your deployment: the round-trip time to the state store (Valkey/Redis), whether the request is translated between shapes, and how often keys fail over. Compare it with relay_request_seconds to see what share of end-to-end latency is Relay’s. Proxy-mode requests (source="proxy") forward the body without translation, so they show the floor for your setup.

Request flow

All are labelled by source, the runner that handled the request: pipeline (normal mode), proxy, ws (a WebSocket frame), batch or count (a count-tokens request).

Metric	Type	Labels	What it answers
`relay_requests_total`	counter	`source`, `status`	How much traffic and how many errors. `status` is the class of the upstream’s response status: `2xx`, `3xx`, `4xx`, `5xx`, or `other` when no upstream answered (refused by routing, a rate limit or an empty key pool; upstream unreachable; client canceled). Requests refused for their credential are counted in `relay_inference_auth_rejected_total` instead, and shed requests in `relay_requests_shed_total`.
`relay_request_seconds`	histogram	`source`	End-to-end latency, from the start of dispatch to the response body closing. Buckets 1 ms to 120 s.
`relay_overhead_seconds`	histogram	`source`	Relay’s own time; see above. Buckets 100 µs to 500 ms.
`relay_admission_seconds`	histogram	`source`	Time from the start of dispatch to the upstream handoff of the attempt that succeeded: routing, rate-limit reservation, key selection, and earlier failed attempts. Observed only when an upstream was reached. Buckets 100 µs to 250 ms.
`relay_post_flight_seconds`	histogram	(none)	Duration of the background work after each response: token extraction, usage, payload and metric recording, rate-limit and host-health commits. It never delays the response.
`relay_inflight_requests`	gauge	`source`	Requests currently open. A streamed request counts until its body closes. Batch items are not counted.
`relay_inflight_limit`	gauge	(none)	The per-pod in-flight cap, `RELAY_MAX_INFLIGHT` (default 2048).
`relay_requests_shed_total`	counter	`stage`	Inference requests refused as overloaded (`503`, or `529` on Anthropic-shaped routes) with `Retry-After: 1` because the pod had no room for them. `stage` is `early` (the in-flight cap was reached; refused before the body is read), `late` (the last in-flight slot was taken after the body was read), or `kv_pool` (`RELAY_REDIS_POOL_MAX_WAITERS` calls were already waiting for a Valkey connection; refused before the body is read). Shed requests write no usage event.
`relay_inference_auth_rejected_total`	counter	`reason`	Inference and OTLP requests refused by the credential check. See Auth rejection reasons.

relay_inflight_requests / relay_inflight_limit is a pod’s headroom before it starts shedding. A non-zero shed rate means the pod is at capacity, which a quota 429 (error_kind rate_limited in usage) is not.

Auth rejection reasons

reason is the message of a 401 or the error code of a 403, exactly as emitted:

Status	`reason` values
`401`	`missing relay key`, `invalid api key`, `api key disabled`, `api key revoked`, `api key expired`, `api key rotated` (an old value past its rotation grace), `invalid token`, `token expired`, `token revoked`, `project unavailable` (a WebSocket frame whose token’s project is gone), `unauthenticated` (anonymous proxy traffic while it is not allowed)
`403`	`passthrough_forbidden`, `proxy_disabled`, `no_policy`, `policy_disabled`, `project_unavailable`

Health and saturation

Metric	Type	Labels	What it answers
`relay_records_lost_total`	counter	`kind` (`usage`, `usage_refused`, `payload`, `payload_refused`)	Background records dropped: the in-memory queue was full, or a ClickHouse write-ahead log went over its segment cap and its oldest segments were deleted unsent (every record in them counts). A `usage` drop is a gap in usage and cost; `usage_refused` and `payload_refused` count dropped records of requests refused before reaching an upstream.
`relay_emit_queue_depth`	gauge	`kind` (`usage`, `usage_refused`, `payload`, `payload_refused`)	Records waiting in the in-memory queue, the leading signal for the drops above. The usage queue holds `RELAY_USAGE_QUEUE_SIZE` (default 16,384), the refused-request usage queue `RELAY_USAGE_REFUSED_QUEUE_SIZE` (default 4,096), the payload queues are bounded by bytes, not records: `RELAY_PAYLOAD_QUEUE_BYTES` (sized from `GOMEMLIMIT`) and `RELAY_PAYLOAD_REFUSED_QUEUE_BYTES` (2 MiB).
`relay_flush_lag_seconds`	gauge	`kind` (`usage`, `payload`)	Age of the oldest unsent write-ahead-log segment of a ClickHouse store. `0` means drained; a rising value means ClickHouse is not accepting writes and the stored view is going stale, before anything is lost.
`relay_audit_dropped_total`	counter	(none)	Admin audit events dropped because the audit queue was full. Each one is a gap in the audit trail.
`relay_provider_keys_down_total`	counter	`reason`	Times a pooled host key was put into cooldown. `reason`: `upstream_auth_failed` (401), `upstream_forbidden` (403), `upstream_rate_limited` (429 with a `Retry-After` over 5 seconds; a shorter one leaves the key in rotation), `upstream_server_error` (5xx), `upstream_network_error` (a failure after the connection was made; a failed dial never cools a key down). See Key pooling.
`relay_kv_errors_total`	counter	`op`, `outcome`	Valkey calls on the inference path that failed or timed out. `op` is the call site: `limit_reserve`, `limit_commit`, `breaker_read`, `breaker_write`, `key_rotation`, `host_health`, `token_ratio`. `outcome` is what Relay did instead: `refused` (the request was rejected), `shed` (`limit_reserve` only: the request was rejected with its rate-limit check unsent because `RELAY_REDIS_POOL_MAX_WAITERS` calls were already waiting for a connection; requests refused for this before their body was read count in `relay_requests_shed_total{stage="kv_pool"}` instead), `fallback` (decided without the stored value, e.g. a key’s breaker read as healthy), or `dropped` (the write was abandoned). Any rate means Valkey is stalling or unreachable.
`relay_ratelimit_unanswered_reserve_releases_total`	counter	`outcome`	Concurrency leases freed for rate-limit checks that Valkey may have run but never answered (the request was refused). `outcome`: `released`, `failed` (the release errored; the lease frees when its window ends), `skipped` (64 releases already in flight on this pod). The request’s other counters are not refunded.
`relay_settle_queue_full_total`	counter	(none)	Served requests whose rate-limit commits found the settle queue full (`RELAY_REDIS_POSTFLIGHT_QUEUE_SIZE`) and went out on a Valkey round trip of their own instead of a shared one. Nothing is lost; a sustained rate means the post-flight writers fall behind Valkey.
`relay_settle_retries_total`	counter	(none)	Queued settles whose shared round trip failed in transport (Valkey unreachable, a dropped connection, a timed-out reply) and were sent once more on a round trip of their own. A commit that had already run is not counted twice. A retry that fails too shows in `relay_kv_errors_total` (`limit_commit`, `dropped`).
`relay_upstream_connections_total`	counter	`kind` (`new`, `reused`)	Upstream connections per attempt: dialed or taken from the keep-alive pool. A `new` rate close to the request rate means connection churn.
`relay_catalog_apply_errors_total`	counter	`kind`	Catalog change notifications that failed to apply incrementally, by entity kind (`provider`, `host`, `model`, `hostkey`, `policy`, …). Each failure falls back to a full catalog reload, so a sustained rate means repeated rebuilds.
`relay_dropped_params_total`	counter	`param`, `model`	Request parameters removed before the upstream call because the routed model’s catalog entry declares them unsupported. The response carries an `X-WR-Warnings` header naming them.
`relay_stream_events_dropped_total`	counter	`adapter`, `direction`, `reason`	Stream events skipped on a translated stream because a translator could not convert them. `direction` is `to_canonical` (an event from the upstream, `adapter` is the upstream shape) or `from_canonical` (rendering into the caller’s shape); `reason` is `syntax`, `type` or `invalid`. The stream continues; if the dropped event was the last one, the stream ends with an error event.
`relay_tokencount_ratio_writes_total`	counter	`outcome`	Token-count calibration samples: `written` to the state store, `skipped` (within 5% of the ratio this pod last wrote, less than 5 minutes ago), or `failed` (the write errored).

relay_provider_keys_down_total counts transitions into cooldown, not keys down right now: breaker state is shared across pods, so a per-pod gauge would disagree between pods, while transition counts add up. Alert on its rate.

OTLP receiver

Counted only for exports to the OTLP receiver.

Metric	Type	Labels	What it answers
`relay_otlp_exports_total`	counter	`signal` (`traces`, `logs`), `result`	Export requests by result: `accepted`, `disabled` (receiver off, 404), `bad_request`, `too_large`, `too_many_records` (more than 10,000 model calls), `unsupported_media`, `policy_disabled` (403), `rate_limited` (429), `backpressure` (the usage queue is half full or lacks room, 503). Exports refused for a missing or invalid credential are counted in `relay_inference_auth_rejected_total`.
`relay_otlp_spans_total`	counter	`outcome`	Spans in trace exports: `recorded`, `duplicate` (already recorded), `ignored` (not a model call), `rejected` (no trace and span id, starting more than 5 minutes in the future, or older than the usage retention). A high `ignored` share is normal for clients that export whole traces.
`relay_otlp_log_records_total`	counter	`outcome`	Log records in logs exports, with the same outcomes. `ignored` is any record other than the `gen_ai.client.inference.operation.details` event.
`relay_otlp_content_total`	counter	`outcome`	Reported calls that carried message content while content capture was on: `stored`, `duplicate`, `dropped` (the payload queue was half full or lacked room), `policy` (the governing policy does not capture payloads), `expired` (older than the payload retention). The usage event is recorded in every case.
`relay_otlp_marker_errors_total`	counter	`op`	Failed state-store operations of the duplicate check. `mark`: calls were stored without the check, so a resend may store them again. `unmark`: calls stayed marked although a queue refused them, so a resend skips them and those records are lost.

Get Started

Build with Relay

How It Works

Deploy and Operate

Observability

Security and Access

Reference

Scrape endpoint

Measuring Relay overhead

Request flow

Auth rejection reasons

Health and saturation

OTLP receiver

​Scrape endpoint

​Measuring Relay overhead

​Request flow

​Auth rejection reasons

​Health and saturation

​OTLP receiver

Scrape endpoint

Measuring Relay overhead

Request flow

Auth rejection reasons

Health and saturation

OTLP receiver