> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wyolet.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Payload logging

> Capture full request and response bodies, choose where they are stored, read them back, and erase them

Payload logging stores the full request and response bodies of a request and attaches them to its [usage event](https://docs.wyolet.com/concepts/usage.md) by request id. Use it to see exactly what a client sent and what the provider returned. It is off by default, and nothing is captured until both an operator switch and the request's policy allow it.

Bodies are stored as Relay received and sent them, after decompression. Streamed responses are captured as the stream of frames the upstream sent. Capture happens after the response, in the background; captured bodies wait in a queue bounded by bytes (`RELAY_PAYLOAD_QUEUE_BYTES`, 1/16 of `GOMEMLIMIT` by default; see [Configuration](https://docs.wyolet.com/configuration.md)), and when it is full new captures are dropped and counted in `relay_records_lost_total{kind="payload"}`. Requests refused before reaching an upstream keep their captures in a small queue of their own (`RELAY_PAYLOAD_REFUSED_QUEUE_BYTES`, 2 MiB), written after the served ones and counted as `kind="payload_refused"` when dropped.

> [!WARNING]
>
> Bodies hold prompts and completions, which are often sensitive. Turn capture on for the policies you need, set a `maxBytes` cap, and set a retention.

## Enabling capture

**1. Turn on the payload-logging section**

`PUT /api/settings/payload-logging` with `enabled: true` and a backend. While `enabled` is false nothing is captured, whatever the policies say.

**2. Opt in on the policy**

Set `payloadLoggingEnabled: true` on the [policy](https://docs.wyolet.com/concepts/policies.md) that governs the traffic. The policy decides, whatever the relay key's own `payloadLoggingEnabled` says. Only a relay key that no policy governs opts in with its own flag.

The governing policy is the one inference resolves: the relay key's own policy, else its service account's, else the first of the project's policy bindings, in priority order, that names the caller. The rule is the same for normal and [proxy-mode](https://docs.wyolet.com/concepts/proxy-mode.md) requests and for WebSocket frames. To capture everything a policy governs, set the flag on that policy. To capture a single key's traffic, give that key a policy of its own with the flag set.

```bash theme={null}
# 1. Turn capture on and pick a backend.
curl -X PUT http://localhost:8081/api/settings/payload-logging \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true, "backend": "clickhouse", "maxBytes": 1048576, "retentionDays": 30,
    "file": {"path": "relay-payloads.jsonl"},
    "s3": {"endpoint": "", "bucket": "", "useSSL": true, "accessKey": {"kind": ""}, "secretKey": {"kind": ""}},
    "clickhouse": {}
  }'

# 2. Opt in on the policy that governs the traffic. PUT replaces the row,
#    so send back the policy as read with GET, with the flag set.
curl -X PUT http://localhost:8081/api/policies/by-id/<id> \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"metadata": {...}, "spec": {..., "payloadLoggingEnabled": true}}'
```

The settings write replaces the whole section: `enabled`, `backend`, `maxBytes`, `retentionDays` and every backend block are required, including the blocks not in use. To change one field, read the section with `GET /api/settings/payload-logging` and send its `value` back with the change. Changes apply without a restart. Only an admin may change this section.

| Field | Default | Meaning |
| - | - | - |
| `enabled` | `false` | The operator switch. |
| `backend` | `file` | `file`, `s3` or `clickhouse`. |
| `maxBytes` | `4194304` (4 MiB) | Cap per stored body, measured on the decompressed body. A longer body is cut and flagged as truncated. `0` stores bodies whole. |
| `retentionDays` | `30` | How long the `clickhouse` backend keeps bodies; `0` keeps them forever. |
| `file.path` | `relay-payloads.jsonl` | File for the `file` backend. |
| `s3.*` | | See [S3](#s3). |
| `clickhouse.walDir`, `clickhouse.dedup` | | See [ClickHouse](#clickhouse). |

## What is never captured

* Requests whose governing policy does not have `payloadLoggingEnabled`, and any request while the section is off.
* Anonymous proxy traffic, which has no relay key.
* Batch items.
* Usage events themselves: `/api/usage/*` and `/api/logs` never carry bodies, only the detail endpoint below does.

Message content reported over OpenTelemetry follows its own switch; see [Message content](https://docs.wyolet.com/concepts/otlp-receiver.md#message-content).

## Backends

### File

JSONL on local disk at `file.path`. It is never pruned, each pod writes and reads its own file, and the read path scans the whole file. Use it for a single node or a trial only.

### S3

One JSON object per request at `<prefix>/<YYYY>/<MM>/<DD>/<request_id>.json` in an S3-compatible bucket (AWS S3, MinIO and others). Expire objects with a bucket lifecycle rule; Relay does not delete them.

| Field | Meaning |
| - | - |
| `endpoint` | `host:port`, no scheme, such as `s3.amazonaws.com`. |
| `bucket` | Required when the backend is `s3`. |
| `region`, `prefix` | Optional. |
| `useSSL` | Must be `true` unless `RELAY_COOKIE_SECURE=false`. |
| `accessKey`, `secretKey` | Secret references, or both empty to use the S3 client's ambient credentials (an IAM role, for example). |

The credentials are references limited to this section: an env reference must name a variable starting with `RELAY_PAYLOAD_S3_` (for example `{"kind": "env", "env": "RELAY_PAYLOAD_S3_ACCESS_KEY"}`), and a stored reference must use an id starting with `payload-logging:`. A stored section that breaks these rules is read as the default (capture off, `file` backend) and logged at startup; rename the variables and write the section again.

### ClickHouse

Bodies are stored in ClickHouse on the connection usage uses (`RELAY_CH_DSN`), buffered in a local write-ahead log first (`clickhouse.walDir`, default `relay-payload-wal`). `retentionDays` becomes the table's TTL and is applied to the existing table whenever the section changes, so shortening it deletes older bodies. This is the only backend that supports [erasure](#erasing-captured-bodies).

#### Deduplication

Agent clients resend the whole conversation on every turn, so most of each request body repeats earlier ones. `clickhouse.dedup: true` (off by default) stores each distinct message, tool list, and large field such as a system prompt once, and each request as the rest of its body plus references to them.

* Bodies read back byte for byte. A body that is not a JSON object, including a truncated one, is stored whole, so pair dedup with `maxBytes: 0`.
* Pieces are shared only between requests of the same project.
* Each request also stores the SHA-256 of its body as received; a body that does not rebuild to that digest is refused with an error rather than returned.
* Response bodies are not deduplicated.
* Rows written before switching either way stay readable.

## Reading captured payloads

The bodies come back on the request log's detail endpoint, under `payload`:

```bash theme={null}
curl -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" \
  http://localhost:8081/api/logs/<request_id>
```

```json theme={null}
{
  "log": {"request_id": "…", "model": "gpt-5", "status": 200, "...": "..."},
  "payload": {
    "request_body": "{\"model\":\"gpt-5\",\"messages\":[…]}",
    "response_body": "{\"id\":\"chatcmpl-…\",…}",
    "request_truncated": false,
    "response_truncated": false
  }
}
```

`payload` is absent when the request was not captured or its body has expired. The same read scope applies as for [usage](https://docs.wyolet.com/concepts/usage.md#reading-usage): a caller who may not see the event gets `404`, and a body is joined only when it was captured for the same project, principal and relay key as the event.

## Erasing captured bodies

An admin can delete the bodies captured for a project, for a principal (a user or service account), or for one principal within one project:

```bash theme={null}
curl -X POST http://localhost:8081/api/logs/erase \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"projectId": "<project id>", "principalId": "<principal id>"}'
```

At least one of `projectId` and `principalId` is required. The response, `{"requests": N}`, counts the requests whose bodies were deleted. The call is recorded in the [audit log](https://docs.wyolet.com/concepts/audit.md).

* The call returns once ClickHouse has rewritten the data that held them, so they can no longer be read; ClickHouse deletes the replaced files from disk after `old_parts_lifetime` (8 minutes by default).
* Usage events (tokens, cost, timing, ids) are kept.
* Bodies a relay has captured but not yet delivered to ClickHouse are not affected; repeat the call after the traffic has stopped.
* With `clickhouse.dedup`, a request of the same project that another relay stores while the erase runs can be left pointing at a deleted piece. Reading it fails with an error, never returns different bytes, until a later request of that project carries the piece again.
* Only the `clickhouse` backend supports erasure, and only for the backend `payload-logging` currently names; `file` and `s3` answer `501`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.