Stored secrets and key rotation

​

How values are encrypted

• One data key per owner. Every stored value belongs to a scope: user:<id> or project:<id> for a host key owned by a user or a project, system for host keys the deployment owns and for Relay’s own secrets. Each scope has its own data key. When a host key changes owner, its value is re-encrypted under the new owner’s data key in the same transaction.
• Values are sealed with XAES-256-GCM (AES-256-GCM with a per-value derived key and a random 192-bit nonce). A sealed value names the data key that sealed it, and its row id is bound in as associated data, so ciphertext copied to another row or another scope does not decrypt.
• Data keys are wrapped by the master key. Data keys live in the secret_data_keys table, each encrypted (wrapped) by a key-encryption key (KEK). The KEK is the master key, RELAY_MASTER_KEY. A wrapped data key is bound to its id and scope.
• A pod can hold more than one KEK. RELAY_MASTER_KEY_EXTRA adds further KEKs, used while changing the master key. Which KEK wraps new data keys, the target, is recorded in the database, not taken from the order of the environment variables; only a rewrap moves it. A pod that does not hold the target KEK never creates a data key.
• Each data key is in one of three states. primary keys encrypt their scope’s new values, one per scope. decrypt keys were replaced by a rotation and still decrypt. destroyed keys keep only their id and scope, so a value found under one later fails as destroyed rather than unknown. One pod at a time rewrites values under a decrypt key to their scope’s primary, in the background.
• Nothing changes on the request path. A pod unwraps the data keys when it starts and keeps resolved credentials in memory. Key changes reach every pod through Postgres NOTIFY within about a second. A pod that missed one still behaves correctly: the database refuses a write under a key that is no longer its scope’s primary, and the pod reloads the key and retries; a value under a key the pod has not seen yet makes it look the key up.
• A missing KEK locks only its data keys. A data key wrapped by a KEK the pod does not hold is reported locked; the values under it do not resolve on that pod (their host keys load unresolved), and the pod keeps serving everything else.

​

What the master key protects against

​

Who can manage keys

​

Checking the key ring

relay secret status
# or
curl https://relay.example.com/api/secret-keys -H "Authorization: Bearer $RELAY_ADMIN_TOKEN"

​

Rotating a data key

relay secret rotate --scope project:<id>     # or --all
# or
curl -X POST https://relay.example.com/api/secret-keys/rotate \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"scope": "project:<id>"}'             # or {"all": true}

1. The new data key becomes the scope’s primary; the old one moves to decrypt and keeps decrypting.
2. Every pod learns of the change within about a second.
3. The running pods rewrite the scope’s values under the new key. The old key’s rows falls to 0. With no server running, relay secret migrate does the rewrite.
4. Optionally, destroy the old key.

​

Destroying a data key

relay secret destroy <data key id> [--force]
# or
curl -X POST https://relay.example.com/api/secret-keys/by-id/<data key id>/destroy \
  -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"force": true}'                       # body optional

​

Changing the master key

1. Add the new key. Generate one with openssl rand -base64 32 (or POST /api/master-key/generate, which only returns a candidate and installs nothing). Set it as RELAY_MASTER_KEY_EXTRA, leave RELAY_MASTER_KEY unchanged, and roll every pod. heldKeks now lists both; targetKek is still the old key.
2. Rewrap. Once every pod holds the new key, run
   relay secret rewrap --to <new kek id>
   # or
   curl -X POST https://relay.example.com/api/secret-keys/rewrap \
     -H "Authorization: Bearer $RELAY_ADMIN_TOKEN" -H "Content-Type: application/json" \
     -d '{"to": "<new kek id>"}'
   
   The new KEK id is the entry in heldKeks that is not targetKek. The target moves to the new key and every data key is wrapped under it; new data keys are wrapped under it from now on. Nothing restarts. The rewrap is refused (409) when the process running it does not hold that KEK, and while legacyRows is above 0. Data keys it cannot unwrap, because their KEK is not on that process, are listed in skipped (the CLI exits non-zero); run it again where both their KEK and the new one are held.
3. Make the new key the master key. When relay secret status shows every data key under the new KEK, set RELAY_MASTER_KEY to the new key and move the old one to RELAY_MASTER_KEY_EXTRA, then roll every pod.
4. Drop the old key. Once every pod runs the step 3 environment, remove RELAY_MASTER_KEY_EXTRA and roll again. Then destroy every copy of the old key.

​

If the master key leaks

1. Rotate every data key: relay secret rotate --all.
2. Change the master key.
3. Wait until every old data key shows 0 rows, then destroy each one, with --force if you cannot wait 24 hours.
4. Rotate every upstream credential stored in Relay at its provider, and store the new values. Re-encrypting a value does not undo its exposure.

​

Upgrading to data keys

1. Back up the affected tables: pg_dump -t secret_values -t secrets -t settings.
2. Scale the deployment to 0 and wait until no relay pod is running. This is the only supported way to stop the pods for this upgrade: with a Recreate update strategy the old pods keep serving while the chart’s migration Job applies migration 34, and every secret they write in that window is refused.
3. Deploy the new version. Migration 34 copies secret_values to secret_values_legacy and adds the data-key tables. With the chart, its migration Job applies it; see the chart README’s “Stop-and-start upgrades”.
4. Scale the deployment back up if the deploy did not. The new pods convert every stored value under its owner’s data key, usually within seconds. relay secret status shows legacyRows reaching 0; pods log a warning while it is above 0. With the servers stopped, relay secret migrate converts synchronously.

​

Rolling back

relay secret downgrade     # rewrites every stored value to the old format under RELAY_MASTER_KEY
relay migrate down 33

BEGIN;
SET LOCAL relay.secret_legacy_write = 'on';
DELETE FROM secret_values;
INSERT INTO secret_values (id, ciphertext, nonce, key_version, created_at, updated_at)
SELECT id, ciphertext, nonce, key_version, created_at, updated_at FROM secret_values_legacy;
COMMIT;

BEGIN;
ALTER TABLE secret_values DISABLE TRIGGER secret_values_check;
DELETE FROM secret_values;
INSERT INTO secret_values (id, ciphertext, nonce, key_version, created_at, updated_at)
SELECT id, ciphertext, nonce, key_version, created_at, updated_at FROM secret_values_legacy;
ALTER TABLE secret_values ENABLE TRIGGER secret_values_check;
COMMIT;

relay migrate down 33