> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wyolet.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Stored secrets and key rotation

> How Relay encrypts the credentials it stores, and how to rotate data keys and change the master key

Relay keeps some credentials encrypted in Postgres: host keys with `valueFrom.kind: stored`, the signing key for [inference tokens](https://docs.wyolet.com/concepts/auth.md#inference-tokens), and stored payload-logging S3 credentials. Host keys that reference an environment variable are not stored and are not affected by anything on this page.

## How values are encrypted

* **One data key per owner.** Every stored value belongs to a scope: `user:<id>` or `project:<id>` for a host key owned by a user or a project, `system` for host keys the deployment owns and for Relay's own secrets. Each scope has its own data key. When a host key changes owner, its value is re-encrypted under the new owner's data key in the same transaction.
* **Values are sealed with XAES-256-GCM** (AES-256-GCM with a per-value derived key and a random 192-bit nonce). A sealed value names the data key that sealed it, and its row id is bound in as associated data, so ciphertext copied to another row or another scope does not decrypt.
* **Data keys are wrapped by the master key.** Data keys live in the `secret_data_keys` table, each encrypted (wrapped) by a key-encryption key (KEK). The KEK is the master key, `RELAY_MASTER_KEY`. A wrapped data key is bound to its id and scope.
* **A pod can hold more than one KEK.** `RELAY_MASTER_KEY_EXTRA` adds further KEKs, used while changing the master key. Which KEK wraps new data keys, the *target*, is recorded in the database, not taken from the order of the environment variables; only a [rewrap](#changing-the-master-key) moves it. A pod that does not hold the target KEK never creates a data key.
* **Each data key is in one of three states.** `primary` keys encrypt their scope's new values, one per scope. `decrypt` keys were replaced by a rotation and still decrypt. `destroyed` keys keep only their id and scope, so a value found under one later fails as destroyed rather than unknown. One pod at a time rewrites values under a `decrypt` key to their scope's primary, in the background.
* **Nothing changes on the request path.** A pod unwraps the data keys when it starts and keeps resolved credentials in memory. Key changes reach every pod through Postgres `NOTIFY` within about a second. A pod that missed one still behaves correctly: the database refuses a write under a key that is no longer its scope's primary, and the pod reloads the key and retries; a value under a key the pod has not seen yet makes it look the key up.
* **A missing KEK locks only its data keys.** A data key wrapped by a KEK the pod does not hold is reported `locked`; the values under it do not resolve on that pod (their host keys load unresolved), and the pod keeps serving everything else.

## What the master key protects against

| An attacker holding | Can read |
| - | - |
| A database dump or backup | No stored secret |
| `RELAY_MASTER_KEY` alone | No stored secret: it only opens data keys, which live in the database |
| The database or a backup, and `RELAY_MASTER_KEY` | **Every stored secret**, including values whose data key was destroyed after that backup was taken |
| The memory of a running pod | Every credential the pod has resolved, and every data key it has unwrapped |
| Write access to the database | No secret by swapping ciphertext between rows or scopes, which fails to decrypt. Pointing a host's base URL or the payload-logging S3 endpoint somewhere else would still make Relay send a credential there; database access control prevents that, not encryption. |

With the master key in an environment variable, whoever holds both the database and that variable can open every scope: per-owner data keys do not isolate tenants from someone holding the master key. That isolation would need a key-encryption key held in a KMS that never releases it; Relay's key-encryption keys are the environment variables above.

What per-owner data keys do give you: one owner's key can be rotated without touching any other, a destroyed data key makes its values unreadable from the live database (backups taken earlier stay readable until the master key changes), and every value is bound to its row.

## Who can manage keys

The control API endpoints below take an admin in either authorization mode: the admin bearer (`RELAY_ADMIN_TOKEN`) or a user carrying the `admin` role. Their RBAC kind is `secret-keys`, with the verbs `list`, `rotate`, `rewrap` and `destroy`; a role binding that grants those verbs is not enough on its own. Every rotate, rewrap and destroy, allowed or refused, is written to the [audit log](https://docs.wyolet.com/concepts/audit.md) with the data key id and scope or the KEK id. Responses carry ids and metadata, never key bytes.

The `relay secret` commands run the same operations directly against Postgres, with `RELAY_PG_DSN` and the master key variables of the deployment, so they work while the control plane is down. Every command except `status` writes the same audit rows, allowed or refused, with the actor kind `cli` and the operating-system user (`$USER`) as its name; `migrate` and `downgrade` write one row each, named by the number of values they converted. If the audit write fails, the command says so and exits non-zero after the change it made.

## Checking the key ring

```bash theme={null}
relay secret status
# or
curl https://relay.example.com/api/secret-keys -H "Authorization: Bearer $RELAY_ADMIN_TOKEN"
```

| Field | Meaning |
| - | - |
| `dataKeys[]` | Every data key: `id`, `scope`, `state`, `kek` (the KEK it is wrapped under), `createdAt`, `demotedAt`, `rows` (stored values under it), `locked` |
| `targetKek` | The KEK new data keys are wrapped under |
| `heldKeks` | The KEKs the answering process holds, `RELAY_MASTER_KEY` first |
| `legacyRows` | Stored values still in the format from before data keys; see [Upgrading](#upgrading-to-data-keys) |
| `undecryptableRows` | Ids of the stored values the answering process cannot decrypt: values in the old format that none of its master keys opens, and values under a data key that is `locked` on it, destroyed, or unknown. A value's id is the id of the host key it belongs to. See [Troubleshooting](https://docs.wyolet.com/troubleshooting.md#the-original-master-key-is-lost) |

A KEK id is 32 hex digits derived from the key; it does not reveal the key. Behind a load balancer, `heldKeks` describes whichever pod answered.

## Rotating a data key

Rotation is online: no restart, no change to the environment.

```bash theme={null}
relay secret rotate --scope project:<id>     # or --all
# or
curl -X POST https://relay.example.com/api/secret-keys/rotate \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"scope": "project:<id>"}'             # or {"all": true}
```

1. The new data key becomes the scope's primary; the old one moves to `decrypt` and keeps decrypting.
2. Every pod learns of the change within about a second.
3. The running pods rewrite the scope's values under the new key. The old key's `rows` falls to 0. With no server running, `relay secret migrate` does the rewrite.
4. Optionally, [destroy](#destroying-a-data-key) the old key.

`all` rotates every scope that has a data key. A scope with no data key is refused (`404`): a scope gets its first key when a value is first stored in it, so the name is usually a typo.

## Destroying a data key

```bash theme={null}
relay secret destroy <data key id> [--force]
# or
curl -X POST https://relay.example.com/api/secret-keys/by-id/<data key id>/destroy \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"force": true}'                       # body optional
```

Destroy is refused (`409`, with the reason) for a scope's primary, while any stored value still uses the key (`rows` above 0), and for 24 hours after the key was demoted. The wait covers a pod that has not seen the rotation yet; `force` skips only the wait, for a key you believe is compromised. The key's row stays as a tombstone. Destroying is optional: a `decrypt` key with no rows does nothing.

## Changing the master key

The master key changes in two rolling restarts and one command. The data keys themselves stay the same, so no stored value is rewritten.

1. **Add the new key.** Generate one with `openssl rand -base64 32` (or `POST /api/master-key/generate`, which only returns a candidate and installs nothing). Set it as `RELAY_MASTER_KEY_EXTRA`, leave `RELAY_MASTER_KEY` unchanged, and roll every pod. `heldKeks` now lists both; `targetKek` is still the old key.
2. **Rewrap.** Once every pod holds the new key, run

   ```bash theme={null}
   relay secret rewrap --to <new kek id>
   # or
   curl -X POST https://relay.example.com/api/secret-keys/rewrap \
     -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" -H "Content-Type: application/json" \
     -d '{"to": "<new kek id>"}'
   ```

   The new KEK id is the entry in `heldKeks` that is not `targetKek`. The target moves to the new key and every data key is wrapped under it; new data keys are wrapped under it from now on. Nothing restarts. The rewrap is refused (`409`) when the process running it does not hold that KEK, and while `legacyRows` is above 0. Data keys it cannot unwrap, because their KEK is not on that process, are listed in `skipped` (the CLI exits non-zero); run it again where both their KEK and the new one are held.
3. **Make the new key the master key.** When `relay secret status` shows every data key under the new KEK, set `RELAY_MASTER_KEY` to the new key and move the old one to `RELAY_MASTER_KEY_EXTRA`, then roll every pod.
4. **Drop the old key.** Once every pod runs the step 3 environment, remove `RELAY_MASTER_KEY_EXTRA` and roll again. Then destroy every copy of the old key.

> [!WARNING]
>
> Do not rewrap before every pod holds the new key. A pod without the target KEK cannot create data keys, so storing the first secret of a new owner fails on that pod until it restarts with the key.

The copy of the stored secrets taken at the [upgrade](#upgrading-to-data-keys), `secret_values_legacy`, stays encrypted under the master key of that time. Keep that key until you no longer need the [last-resort rollback](#rolling-back).

## If the master key leaks

Changing the master key alone is not enough: whoever has the old key and the database may already have unwrapped the data keys.

1. Rotate every data key: `relay secret rotate --all`.
2. [Change the master key](#changing-the-master-key).
3. Wait until every old data key shows 0 `rows`, then destroy each one, with `--force` if you cannot wait 24 hours.
4. Rotate every upstream credential stored in Relay at its provider, and store the new values. Re-encrypting a value does not undo its exposure.

Treat database backups taken before step 3 as exposed: they still open with the old master key.

If a single data key may have leaked, rotate its scope, wait for its `rows` to reach 0, destroy it, and rotate the provider credentials stored in that scope.

## Upgrading to data keys

The release that introduces data keys (schema version 34) converts every stored value once. The upgrade is a stop-and-start, not a rolling update: from schema 34 the database refuses values in the old format, so old pods that keep serving fail their secret writes and cannot read converted values.

1. Back up the affected tables: `pg_dump -t secret_values -t secrets -t settings`.
2. Scale the deployment to 0 and wait until no relay pod is running. This is the only supported way to stop the pods for this upgrade: with a `Recreate` update strategy the old pods keep serving while the chart's migration Job applies migration 34, and every secret they write in that window is refused.
3. Deploy the new version. Migration 34 copies `secret_values` to `secret_values_legacy` and adds the data-key tables. With the chart, its migration Job applies it; see the chart README's "Stop-and-start upgrades".
4. Scale the deployment back up if the deploy did not. The new pods convert every stored value under its owner's data key, usually within seconds. `relay secret status` shows `legacyRows` reaching 0; pods log a warning while it is above 0. With the servers stopped, `relay secret migrate` converts synchronously.

### Rolling back

Stop every pod (or set `RELAY_MIGRATE_ON_BOOT=off` on them first), then with the new binary:

```bash theme={null}
relay secret downgrade     # rewrites every stored value to the old format under RELAY_MASTER_KEY
relay migrate down 33
```

Then deploy the previous image. `relay secret downgrade` needs `RELAY_MASTER_KEY` set to the key the previous version will use, and keeps secrets written since the upgrade, including a rotated token signing key. `relay migrate down` below 34 refuses while any value is still in the new format.

As a last resort, when `relay secret downgrade` cannot finish (it lists the values it could not decrypt), restore the copy taken at the upgrade and then roll the schema back:

```sql theme={null}
BEGIN;
SET LOCAL relay.secret_legacy_write = 'on';
DELETE FROM secret_values;
INSERT INTO secret_values (id, ciphertext, nonce, key_version, created_at, updated_at)
SELECT id, ciphertext, nonce, key_version, created_at, updated_at FROM secret_values_legacy;
COMMIT;
```

If the `INSERT` fails with `dek_id does not match the sealed header` (SQLSTATE `WSK02`), an old-format ciphertext happens to begin with the bytes that mark the new format (about one value in four billion), and the transaction changed nothing. Run the restore with the check disabled for its duration instead; this needs the table owner's role, and the transaction reverts the trigger state if anything fails:

```sql theme={null}
BEGIN;
ALTER TABLE secret_values DISABLE TRIGGER secret_values_check;
DELETE FROM secret_values;
INSERT INTO secret_values (id, ciphertext, nonce, key_version, created_at, updated_at)
SELECT id, ciphertext, nonce, key_version, created_at, updated_at FROM secret_values_legacy;
ALTER TABLE secret_values ENABLE TRIGGER secret_values_check;
COMMIT;
```

```bash theme={null}
relay migrate down 33
```

> [!WARNING]
>
> The restore discards every stored secret written since the upgrade. Re-enter the values of stored host keys created or changed since then, and, if `signingKey` in the `auth:tokens` settings section names a secret that no longer exists, clear it so the next boot generates a new signing key (inference tokens signed by the lost key stop verifying).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.