Rotate secrets

Before you start#

• Every secret lives in your secret manager and reaches the services as a file where the service accepts one (platform security: secrets). Rotating means writing a new file and rolling the service that reads it.
• You can roll each deployment without a gap: readiness-gated rolling replacement, ideally maxUnavailable: 0 with a surge. Each service reads its environment only at start.
• You hold a platform administration credential for Access and know which gateways, Data deployments, consoles and scripts present each Access credential.
• You know whether any Data route still uses the legacy v4 identity format; if so, leave it before rotating the gateway's secrets.

What each rotation costs:

Steps#

The secrets are mostly independent, but the order below never leaves one service holding a value its peer does not yet accept: for every shared secret, the side that verifies or accepts it changes first, the side that presents it second, and the old value goes last.

1. Gateway session secret#

It signs the gateway's login state cookie and does not affect the identity signature (gateway operations: rotating the session secret).

1. Write the new value to the secret store; it must be the same on every replica.
2. Roll all replicas onto it quickly, with a surge or in a maintenance window. Replicas holding different secrets cannot read each other's login state, so anyone signing in during the rollout must start again.
3. Remove the old value.

The gateway's rollback notes say sessions survive only while this secret is unchanged (gateway operations: rollback), so expect callers to sign in again after the rotation.

2. Gateway identity signing key#

The gateway signs each request's identity with this P-256 private key; upstreams and Data hold only the public key (upstream identity: key).

bun run identity:key                     # a new pair
GATEWAY_IDENTITY_SIGNING_KEY=<the pem>   # the new private key, or GATEWAY_IDENTITY_SIGNING_KEY_BASE64
bun run identity:key --public            # the public key to hand to upstreams

1. Give every upstream the new public key and have it accept the old and new keys for the duration of the switch (gateway operations: rotating the identity key).
2. Roll the gateway quickly: until the rollout ends, some replicas sign with the old key and some with the new one.
3. Update Data. A Data binding names exactly one publicKeyFile and never falls back to another key (Data spec §4.1), so Data verifies with one key at a time: write the new public key into the binding's file and restart Data right after the gateway rollout (Data operations: rotating secrets). The docs describe no way for a Data binding to accept two keys at once, so requests signed with the other key get 401 UNAUTHENTICATED until both sides match; keep that window short or schedule it.
4. Once every replica signs with the new key, remove the old public key from the upstreams.

3. Access state secret#

It signs and encrypts the login state cookie between Access and the tenant's provider (Access spec §5.3). Generate at least 32 bytes, set the same value on every replica, and roll. Logins in flight end with 400 login_state_invalid and must start again; replicas with different values cannot read each other's cookies, so roll quickly (Access operations: rotating secrets).

openssl rand -base64 32   # store it in the file ACCESS_STATE_SECRET_FILE names

4. Access encryption key#

This key encrypts every tenant's provider and gateway client secrets in the database. Replacing it without re-encrypting makes them unreadable (Access spec §4.3).

1. Generate the new key with openssl rand -base64 32.
2. Re-encrypt both columns in one transaction. The script prints how many tenants it changed, never a secret, and changes nothing if it fails:

   ACCESS_DATABASE_URL=postgres://… ACCESS_SECRET_ENCRYPTION_KEY=the-old-key \
   ACCESS_NEW_SECRET_ENCRYPTION_KEY=the-new-key bun run scripts/cli/rotate-secrets.ts

1. Set the new key as ACCESS_SECRET_ENCRYPTION_KEY and roll at once: replicas still holding the old key cannot read the re-encrypted secrets.
2. Run scripts/cli/admin-smoke.ts.

5. Access signing key#

Access signs its id and access tokens with this RSA key and publishes the public half at /oauth/jwks (Access spec §4.2).

1. Generate a key: openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048.
2. Move the current key and kid to ACCESS_PREVIOUS_SIGNING_KEY and ACCESS_PREVIOUS_SIGNING_KID, set the new key and a new ACCESS_SIGNING_KID, and roll. The JWKS now lists both.
3. Wait at least five minutes, the lifetime of the tokens the old key signed, then remove both ACCESS_PREVIOUS_SIGNING_* variables and roll.

curl -sS https://access.example.com/oauth/jwks | jq -r '.keys[].kid'   # both kids during step 2

Changing the key without keeping the previous one published breaks every live session (Access troubleshooting: token and JWKS problems).

6. Access administration and decision tokens#

Both rotate with a previous-value variable, so nothing is refused until the last step (Access spec §4.3.1):

1. Copy the current values into ACCESS_PREVIOUS_ADMIN_TOKEN and ACCESS_PREVIOUS_DECISION_TOKEN; roll Access.
2. Change the holders: every gateway's GATEWAY_ACCESS_TOKEN (roll each gateway), and PLATFORMCTL_ADMIN_TOKEN wherever operators and scripts use the platform token. Then set the new ACCESS_ADMIN_TOKEN and ACCESS_DECISION_TOKEN; roll Access.
3. When every holder uses the new values, remove both ACCESS_PREVIOUS_* variables; roll Access.

In production each token is at least 16 bytes. Once every gateway presents tenant credentials, retire the shared decision token instead of rotating it: remove ACCESS_DECISION_TOKEN and ACCESS_PREVIOUS_DECISION_TOKEN together, since the previous value alone stops the start (Access operations: requirements).

7. Credentials in the Access maps#

ACCESS_ADMIN_TOKENS (administration) and ACCESS_CONTEXT_TOKENS (the access context read by Data and the console) have no previous-value variable, but each map holds as many tokens as you list (Access configuration: administration credentials). Rotate an entry in three rolls:

1. Add the new token next to the old one, with the same scope; roll Access. ACCESS_ADMIN_TOKENS has no _FILE form, so it is set in the environment.
2. Give the new token to its holder and restart it:

1. Remove the old token from the map; roll Access.

For Data, Access must accept the new token before Data presents it; until the two agree, Data answers 503 AUTHORIZATION_UNAVAILABLE (Data operations: rotating secrets). A tenant credential's audit operator name stays the same across the rotation if you keep its scope unchanged.

8. Tenant client secrets#

Each tenant's gateway, and a console that signs in as that tenant's client, presents a client secret at Access's token endpoint. When the tenant registration receives a new clientSecret, Access keeps accepting the replaced one for 24 hours (Access spec §4.3.1).

1. Send the tenant registration again with the new clientSecret and every other required field, as in Onboard a tenant (API: register a tenant). From now on both secrets are accepted.
2. Set GATEWAY_OIDC_CLIENT_SECRET (or CONSOLE_CLIENT_SECRET) to the new value and roll the gateway or console, well inside the 24 hours.
3. After 24 hours the old secret stops working on its own.

Update the tenant first: the 24-hour window protects the replaced secret, not the new one (Access spec §4.3.1). Re-sending the secret already in use is not a rotation and does not reopen the window.

The tenant's provider secret (idpClientSecret, Access's own secret at the tenant's provider) is changed with the same call. Whether the provider accepts two secrets during the switch is up to the provider; the docs do not cover it. Access drops the tenant's provider client when the registration changes, so the next login uses the new secret (Access spec §6.6).

9. Data cursor key and database credentials#

The gateway public key in Data's bindings is covered in step 2.

10. Console credentials#

The console keeps sessions in process memory, so a restart signs everyone out anyway (console operations: rotating secrets).

11. Everything else#

Check that it worked#

• Every service answers GET /readyz with 200 after its roll; platformctl status reads them all at once, and platformctl doctor checks the credentials in your profile.
• bun run identity:key --public gives the same public key on every gateway replica, and a request through the gateway reaches Data without 401 UNAUTHENTICATED.
• The Access JWKS lists only the new kid once step 5 is finished.
• ACCESS_URL=$ACCESS ACCESS_ADMIN_TOKEN=<new token> bun run scripts/cli/admin-smoke.ts passes, and the old administration token gets 401 after step 6.
• A fresh browser sign-in through each tenant's gateway completes, and access_logins_total{result!="success"} does not rise.
• access_admin_requests_total{status="401"} stays flat after you remove an old token: a holder you forgot would show up there.

If something goes wrong#

Related#

• Platform security
• Gateway operations: secrets
• Access operations: rotating secrets
• Data operations: rotating secrets
• Console operations: rotating secrets
• Respond to an incident
• Upgrade and roll back