/v1/batches takes a list of requests, returns at once, and runs the requests in the background on Relay’s own worker pool. Each item goes through the same routing, policy, rate limits and host-key pool as a live request, and is recorded as a usage event with source=batch.
Relay runs every item itself; it does not use a provider’s batch API, so batch items are billed at the provider’s normal rates.
Submit a batch
The answer is
202 Accepted with the batch id. Leave stream unset in the items: the response is stored whole, not streamed.
Items must match the upstream’s shape
Relay does not translate batch items. An item runs only when the routed model’s host binding uses the same adapter asshape: openai items need an adapter: openai binding, anthropic items an adapter: anthropic binding, openai_responses items an adapter: openai_responses binding.
An item routed to a binding of another shape fails with a cross-shape dispatch error. One exception fails differently: an openai_responses item routed to an adapter: openai binding on the openai host is sent to the Chat Completions endpoint and fails with the upstream’s status. Canonical (/v1/generate) bodies and embeddings requests cannot be batched. To mix providers, submit one batch per shape.
Check progress
RelayKeyHash) and the submitter’s attribution (PolicyID, ProjectID, TeamID, PrincipalKind, PrincipalID, CredentialKind, CredentialID).
Track progress with counts and items: Status is queued from submit until the batch is cancelled, when it becomes cancelled. It does not change as items run. A batch is finished when every item is in a final state (completed, discarded or cancelled).
Collect results
idx is the item’s position in requests. A completed item carries the upstream’s response body in the item’s own shape. A failed item carries the error text only; the upstream’s error body is not kept.
An item fails when:
- the upstream answers with a non-
2xxstatus, or every key of the chosen host fails; - routing rejects it (unknown model, model not granted by the policy, no binding), or its shape does not match the binding;
- the policy’s rate limit is spent when it runs. Items do not wait for the limit to refill;
- the submitting credential stopped working: the relay key was disabled, revoked, expired or rotated out, or the inference token expired or was revoked;
- the policy the batch was submitted under was deleted or disabled;
- it runs longer than 5 minutes.
Cancel
cancelled. An item running on the replica that receives the cancel is stopped; an item already running on another replica finishes.
Access, policy and limits
A batch authenticates like any inference request, with a relay key or an inference token. The policy that governs the credential at submit time is recorded on the batch and applies to every item: its model grants, its rate limits, and its host-key pool, key selection and circuit breakers. The credential itself is checked again when each item runs. Only the submitter can read or cancel a batch: the same relay key, or, for an inference token, any token of the same user. Anyone else gets404. Because access follows the relay key itself, a rotated key cannot read the batches its previous value submitted.
Usage and logging
Every item produces a usage event withsource=batch, attributed to the submitting credential, its project and team, and the batch’s policy, and priced at the binding’s normal rates. Filter with source=batch in the usage API. Item request and response bodies are not captured by payload logging; the results endpoint is where a batch’s responses live.
Running batches on more than one replica
Relay keeps batch state in Postgres but stores each item’s request and response as files underRELAY_BATCH_PAYLOAD_DIR (default relay-batch-payloads, relative to the working directory). Any replica may pick up any item, so every replica must see the same directory. Mount it from a volume all replicas share, such as a ReadWriteMany volume on Kubernetes.
When the directory is not shared, an item picked up by a replica other than the one that received the submission fails with a load input error, and a result written by one replica comes back without response when another replica serves the results request. The Helm chart’s default puts the directory on storage local to each pod, so with more than one replica, give it a shared volume or run a single replica for batch traffic.
Files are not removed when a batch finishes; size the volume for the batches you keep.