relay_overhead_seconds and the metrics under Check a sizing.
Planning figures
Two numbers per box size. The latency knee is the highest load at which p99 relay overhead stays at or under 15 ms with at most 0.1 % errors. Error capacity is the highest load with at most 0.1 % errors, whatever the latency; its p99 is in brackets.
A later run on the v0.11.8 code reached 40,000 rps on 16 cores and 52,000 on 32 cores at error capacity; see the later run on the benchmarks page.
- Size for the latency knee when p99 overhead matters to your callers, and for error capacity when only throughput does.
- Several 8-core pods carry more per core than one large pod. Scale out with replicas rather than up.
- A small request cost about 0.29 ms of relay CPU in this mix. Translation between wire shapes, large bodies, streaming and payload logging all cost more; measure with your own traffic.
- Streaming costs about 213 KiB of working set and 0.45 cores per 1,000 open streams, on top of a base of about 185 MiB.
In-flight cap
A process refuses inference requests as overloaded,503 (529 on Anthropic-shaped routes) with Retry-After: 1, before their body is read, once the requests in flight reach its in-flight limit; a stream holds its slot until it ends. Two limits set it:
- The hard cap, for memory.
RELAY_MAX_INFLIGHT; unset, as many requests as fit in 40 % ofGOMEMLIMITat 256 KiB each, at least 2048: 2048 at the chart’s defaultGOMEMLIMITof 900 MiB, 6553 at 4 GiB, 2048 without a memory limit. - The adaptive limit, for CPU. Every 100 ms the process reads its CPU time as a share of
GOMAXPROCS, and its scheduler latency p99, how long runnable goroutines waited for a CPU. When three windows in a row are at 90 % CPU or aboveRELAY_INFLIGHT_SCHED_P99(default20ms), the limit drops to 90 % of what is in flight; otherwise it grows back by a tenth per window up to the hard cap. A request waiting on its upstream uses no CPU, so long generations don’t lower it; a CPU-saturated process sheds whatever its upstream latency.
RELAY_MAX_INFLIGHT when the derived cap is lower than your traffic needs and memory allows more. Requests in flight are the request rate times how long each request takes, upstream time included: 1,000 rps of 4-second generations needs about 4,000 slots, plus headroom for upstream slowdowns. Small requests cost about 0.2 MiB of working set each above a base of 185 MiB; requests with Claude Code-size bodies (several hundred KB held for the whole stream) fit about 1,000 per GiB. A cap of 2048 with short requests costs no throughput at 8 cores and below; on 16 and 32 cores it held the box near 30,000 rps while cores sat idle, so raise the cap or GOMEMLIMIT there. On Kubernetes set it through relay.extraEnv; the chart has no dedicated value.
Memory and garbage collection
- Set
GOMEMLIMITto about 90 % of the container memory limit. The chart sets it fromgoMemLimit(default900MiB, against a1Gilimit inresources.limits.memory); raise the two together. RELAY_MAX_CONNSdefaults to as many connections as fit in 40 % ofGOMEMLIMIT, so it follows a raised limit without being set. Connections past it wait up toRELAY_CONN_QUEUE_TIMEOUTin a queue ofRELAY_CONN_QUEUE, and the rest are refused with503, so clients that open a new connection per request get an answer they can retry instead of a stalled connect.- Relay runs with
GOGC=400whenGOGCis unset (chart valuegoGC). On 16 and 32 cores it used 9–16 % fewer cores per request than Go’s default of100and cut p99 overhead by a quarter to a half, for 0.7–1.6 GiB more working set. KeepGOMEMLIMITset, so the heap still has a bound; on a pod short of memory, setGOGC=100. - The in-memory state backend (
RELAY_STATE_BACKEND=memory) is for single-pod use; under load it is GC-bound.
Upstream connections
RELAY_UPSTREAM_MAX_IDLE_PER_HOST is how many idle keep-alive connections Relay keeps per upstream host. It defaults to the in-flight cap, so every request in flight can return its connection to the pool. Below the number of requests in flight to one host, connections past the cap close after each response and the next requests dial again: at 128, an 8-core relay past its capacity spent 14.6 % of its CPU dialing and served 14,700 rps at 28,000 offered, against about 22,000 at 1,024. Leave it at the default unless an upstream limits connections.
State store
WithRELAY_STATE_BACKEND=redis, each request made about 18 store commands in the benchmark mix: the rate-limit reserve and settle, key breaker reads and the rotation counter. Before the upstream call, the caller’s limits are reserved in one script call per counter group, the key pick reads the candidates’ breaker records, and the picked key’s upstream limits take one more script call per group. The settle runs after the response, one store call per counter group the request reserved.
- Dragonfly spreads keys over threads, one per core it is given (
--proactor_threads). Run it with--cluster_mode=emulated --lock_on_hashtags: Relay’s keys carry hash tags, and with these flags each script stays on one shard. With default flags every script runs as a transaction across shards and throughput levels off early. With the flags, a 16-core relay was still under 0.1 % errors at 28,000 rps. - Several relays on one store wait longer for it. At the same load per relay, four 8-core relays sharing one Dragonfly waited about ten times longer for a store connection than one relay alone. Plan on about 15,000 rps per 8-core relay at error capacity, or 8,000 at the latency knee, and watch
relay_admission_seconds.
See Configuration for the full list.
Host settings
The benchmark boxes ran with these limits, which a relay box at tens of thousands of requests per second needs:Load balancer
In the four-relay benchmark HAProxy 3.2 sat in front of the relays with keep-alive on both sides andhttp-reuse always, so requests reuse established connections to the relays instead of opening one each. It used about 0.07 cores per 1,000 rps over plain HTTP. In a container, set HAProxy’s nbthread from the CPU limit: it counts the CPUs it may run on, not the container’s CPU quota.
Check a sizing
See Metrics for the full list.