Skip to main content
A release can add Postgres migrations. Relay applies pending ones at boot (RELAY_MIGRATE_ON_BOOT, on by default) under a Postgres advisory lock, so several replicas can start together. relay migrate up applies them as a step of its own, and the Helm chart runs it in a pre-upgrade Job before any pod rolls (see Migrations). Most releases upgrade in place: deploy the new image, or run helm upgrade with the new chart version. The releases under Release notes need extra steps; read the note for every release between your version and the target.

​
Upgrade procedure

  1. Back up Postgres. A rollback is a down-migration that drops what the up-migration added, not a configuration switch.
  2. Read the release notes for every release you cross.
  3. Deploy the new version. When a note calls for it, stop every old pod first, as in Stop-and-start upgrades.

​
Stop-and-start upgrades

Some upgrades must not run old and new pods side by side. For those, scaling to 0 is the supported way to stop the old pods:
  1. kubectl -n relay scale deploy/relay --replicas=0, and wait until kubectl -n relay get pods lists no relay pod. With Argo CD self-heal, pause automated sync first, or it restores the replica count. With Docker, stop every relay container.
  2. Upgrade. The chart’s migrate Job applies the migrations while no pod is running.
  3. Once the upgrade has finished, check that the Deployment is back at its replica count, and scale it up if not.
The chart’s updateStrategy.type: Recreate is not enough: the old pods keep serving while the pre-upgrade migrate Job runs.

​
Lock timeout

Each migration waits at most RELAY_MIGRATE_LOCK_TIMEOUT_S seconds (default 10) for a table lock. An ALTER TABLE queued behind a long transaction would otherwise hold every later query on that table behind it, sign-ins on users included, for as long as the transaction runs. A migration that gives up fails with canceling statement due to lock timeout, applies nothing, and leaves the schema version clean, so boot, relay migrate up, or the chart’s migration Job can be run again once the blocking transaction has ended. The timeout applies to the migration connections only, not to the pool that serves requests; 0 waits indefinitely, and a lock_timeout set in RELAY_PG_DSN takes precedence.

​
Rolling back

  1. Stop every pod: scale the deployment to 0, or set RELAY_MIGRATE_ON_BOOT=off on it first. Every relay runs the pending up-migrations at boot, so a pod that restarts mid-rollback re-applies what you just undid.
  2. With the new binary and RELAY_PG_DSN set, run relay migrate down <target-version>, where the target is the schema version of the release you return to (each release note names it). This runs the bundled down-migrations in reverse order down to, not including, the target; relay migrate down 0 unwinds all of them. A target above the current schema version is refused rather than run as an upgrade.
  3. Deploy the previous image and drop RELAY_MIGRATE_ON_BOOT again, so the next upgrade migrates on boot.
relay migrate down is destructive: it drops the tables and columns its down-migrations remove, with the rows in them. Take the backup before the upgrade, not after.
relay migrate down and relay migrate force read and write relay’s version table, schema_migrations. The auth module’s migrations, which boot and relay migrate up run after relay’s, record their version in auth_schema_migrations. relay migrate force never changes it; relay migrate down to a version below 26 first steps the auth module back to its version 1, because relay’s down-migrations below 26 drop tables the module’s later migrations extend. The next upgrade reapplies them: accounts keep their primary sign-in identity, and further identities and external ids are gone.

​
Rolling back the auth module

The auth module’s migrations only add: the identities table, external_source / external_id on users and groups, scim_clients, and users.resource_version. A binary from before them ignores all of it, and a trigger copies the sign-in identity such a binary writes into identities, so the safe default is to roll back the image and leave the schema where it is. Run the module’s down-migrations only when you need the schema itself back:
It runs the module’s down-migrations to (not including) <version>, with RELAY_PG_DSN set, and leaves relay’s own version alone; a target at or above the module’s current version does nothing. The lowest target is 1, the module’s first migration, whose tables are relay’s own; relay migrate down removes those. Stop the pods or set RELAY_MIGRATE_ON_BOOT=off first, as above, since boot reapplies the module’s migrations.

​
Recovering from a dirty schema version

If a migration is interrupted, the schema is recorded as dirty at that version and every later migrate (the chart’s Job, or a pod migrating on boot) fails with Dirty database version N. To recover:
  1. Read the state: SELECT version, dirty FROM schema_migrations. With the chart’s bundled Postgres: kubectl -n relay exec -it sts/relay-postgresql -- psql -U relay relay -c 'SELECT version, dirty FROM schema_migrations'.
  2. Open migrations/postgres/<N>_*.up.sql at the tag of the release you are upgrading to and check that every object it creates or alters (tables, columns, indexes, constraints, functions, triggers) is present in the database. Each file runs in a single transaction, so it is normally all there or none of it.
  3. All present: record N as applied with relay migrate force N. None present: record the previous version with relay migrate force <N-1>. Partly present: repair by hand or restore the backup before forcing anything. relay migrate force clears the flag without running any migration.
  4. Run the upgrade again; whatever is still pending is applied.
relay migrate force needs only RELAY_PG_DSN. With the chart’s bundled Postgres, forward it and run the release image locally:

​
Release notes

​
0.11.6: data keys for stored secrets (schema 34)

Stored secrets move from the master key to per-scope data keys wrapped by it. The new pods rewrite every stored value, and from migration 34 on the database refuses the old format, so old pods still serving would fail their secret writes and stop reading rewritten values. This is a stop-and-start upgrade. The backup, the conversion check (relay secret status reports legacyRows reaching 0, usually within seconds of the new pods starting) and the rollback (relay secret downgrade, then relay migrate down 33, then the 0.11.5 image) are in Upgrading to data keys. If RELAY_MASTER_KEY held the wrong key at this upgrade, see Wrong master key at the upgrade to data keys.

​
0.10.0: teams, projects and RBAC (schema 25 to 30)

This release adds teams, projects, roles and relay-signed inference tokens, and changes how relay keys are stored. A fleet mixing the old and new binary is only partly functional, so stop every old pod before deploying it. Before you upgrade. Run 0.9.25 first. It boots against a schema newer than its own migrations (it logs a warning and skips migrating) instead of failing at startup. That lets an older pod restart while the upgrade runs, and lets you start the previous image during a rollback. Earlier 0.9.x releases refuse to boot once the new migrations have run. What changes on disk. Migrations 000025 to 000030 add: Rolling upgrades. If you roll pods one at a time, the first new pod migrates the schema while old pods are still serving. Until the last old pod is replaced, creating a key on an old pod fails: the old binary writes a key with no principal, which the new schema rejects. What behaves differently afterwards.
  • Keys without a policy. A key that had no policy before the upgrade lands in the legacy project with no policy bound, and keeps resolving as it did: through the policy-less flow, which the allowMissingPolicy field of the inference settings section governs under RELAY_AUTHZ=single and which RELAY_AUTHZ=rbac refuses. Once you add a policy binding to the legacy project, its keys follow the same rule as every other project: they resolve a policy through their service account or a binding, or are refused.
  • Rate limits. Rate-limit counters are keyed differently, so every bucket starts empty at the cutover. A bare tokens meter now counts input plus output tokens; before the upgrade it summed every token sub-meter, including cache reads, cache writes, and reasoning tokens. To cap those, add rules on the specific meter (tokens.cache_read, tokens.reasoning, …).
  • Multi-user deployments. RELAY_MULTI_USER is read as RELAY_AUTHZ=rbac. Under multi-user, any signed-in user could read the usage and request logs of their own keys; under rbac those reads need a RoleBinding that grants them, for example the built-in developer role (usage and logs) or viewer (usage) on the user’s project. Bind those roles before switching users over, or non-admin users see no usage.
Rolling back. Follow Rolling back with relay migrate down 24, then deploy 0.9.25. A key that resolved its policy through its service account or a policy binding gets that policy written onto the key, the only place the older binary reads it from. migrate down is for rollback, not a round trip: upgrading again runs the backfill afresh, keys that had no user are moved onto newly generated service accounts, and anything created against the newer schema (teams, projects, roles, bindings, service accounts) is gone.