Respond to an incident

Before you start#

Have these ready before an incident, not during one:

export ACCESS=https://access.example.com
export ACCESS_ADMIN_TOKEN=…           # platform credential
export CONTROL=http://127.0.0.1:9090  # gateway-control
export CONTROL_TOKEN=…                # scope: operate

With Access as the gateway's provider, the subject the gateway knows a caller by is the principal UUID Access issues as sub (Access spec §7.1). Look it up first; every step below uses it:

USER_ID=$(curl -sS "$ACCESS/v1/admin/users?tenant=acme&email=alice@acme.example" \
  -H "authorization: Bearer $ACCESS_ADMIN_TOKEN" | jq -r '.principals[0].id')

Steps#

1. Cut off one account#

Work from the source of truth outwards: stop new decisions in Access, then clear what the gateway remembers.

1. Disable the principal. A disabled principal is refused at login (403 principal_disabled) and denied at every decision, whatever grants it holds (Access spec §8.3):

   curl -sS -X POST $ACCESS/v1/admin/users \
     -H "authorization: Bearer $ACCESS_ADMIN_TOKEN" -H "content-type: application/json" \
     -d '{"tenant":"acme","externalSubject":"00u1a2b3c4d5","status":"disabled"}'

   platformctl access principals create --tenant acme --subject 00u1a2b3c4d5 --status disabled

1. Revoke its grants if the account itself is legitimate but its rights were abused or wrongly given. Revoked rows stay for the investigation (Manage access shows the loop that revokes every live grant).
2. Purge the gateway's cached decisions for the subject, so the next request asks Access instead of reusing an allowance for up to GATEWAY_ACCESS_CACHE_TTL_MS:

   curl -sS -X DELETE -H "authorization: Bearer $CONTROL_TOKEN" "$CONTROL/v1/decisions?subject=$USER_ID"

   platformctl sessions purge-decisions --subject "$USER_ID"

1. End the gateway sessions of the subject, without signing anyone else out (gateway operations: revoking sessions):

   curl -sS -X DELETE -H "authorization: Bearer $CONTROL_TOKEN" "$CONTROL/v1/sessions?subject=$USER_ID"

   platformctl sessions revoke --subject "$USER_ID"
   # or, against the shared state directly:
   GATEWAY_SESSION_URL=redis://127.0.0.1:6379/0 bun run sessions -- --subject "$USER_ID"

Each form reports how many sessions it removed. On Redis, sessions created before the subject index existed are not found and only expire on their own; on Postgres, postgres:migrate indexes them.

What can still run after these steps:

On routes where these windows are too long, set accessCacheTtlMs: 0 and accessStaleGraceMs: 0 in the route table and publish it with platformctl routes apply (gateway configuration: route-level overrides).

2. Cut off a whole tenant#

1. Disable the tenant. A disabled tenant refuses every login and denies every decision (Access spec §3.1). Send its registration again with "enabled": false, carrying the required fields as in Onboard a tenant.
2. Purge cached decisions for each of the tenant's workspaces: DELETE /v1/decisions?workspace=<id>, or platformctl sessions purge-decisions --workspace <id>.
3. End its sessions. In the supported topology the tenant has a gateway deployment of its own (Access spec §1.4), so you can end every session of that gateway by deleting its key prefix in the shared state (SCAN and DEL, or FLUSHDB when the prefix owns the database), once per prefix (gateway operations: revoking sessions).
4. Withdraw its credentials. Remove the tenant's entries from ACCESS_ADMIN_TOKENS, drop its UUID from every list in ACCESS_CONTEXT_TOKENS, and remove it from CONSOLE_TENANT_ADMIN_TOKENS; roll Access and the console.
5. Close its Data workspaces if data must stop being served at once: bun run workspace -- disable --id <uuid>, run as the schema owner, makes the workspace answer 404 (Data operations: workspaces).

3. Replace the secrets that may have leaked#

Rank the leak by what it lets an attacker do, then follow the matching step of Rotate secrets. The signing keys are the trust boundaries between the services (platform security: trust boundaries).

The rolling procedures keep the old value valid for a while to avoid an outage. For a leaked value, shorten or skip that window and accept the refusals it causes:

• For the administration and decision tokens, you can set the new value without first copying the old one into ACCESS_PREVIOUS_*; every holder you have not updated yet is refused until it is. A gateway refused by Access answers its callers 503 access_unavailable.
• For a map entry, remove the old token in the same roll that adds the new one.
• For a tenant client secret, Access keeps the replaced secret valid for 24 hours, and the docs describe no way to end that window early. If the old secret must stop working now, disabling the tenant (step 2) is the documented hard stop.

4. Find out what happened#

Start from what you know (a request id, a subject, a time window) and widen out.

1. The Access audit lists every administration change with its operator, address and request id. Filter by tenant or action and keep an export of the window (Access spec §10.15):

   curl -sS "$ACCESS/v1/admin/audit?tenant=acme&action=grant.created&limit=200" \
     -H "authorization: Bearer $ACCESS_ADMIN_TOKEN"
   curl -sS "$ACCESS/v1/admin/audit?format=jsonl&tenant=acme" \
     -H "authorization: Bearer $ACCESS_ADMIN_TOKEN" -o acme-audit.jsonl

   platformctl access audit export --tenant acme --format jsonl --out acme-audit.jsonl

An operator shown as twelve hex characters is a credential with no name; give operators named tenant credentials so the next audit names a person (Access operations: hardening). The grants list shows who granted each right (grantedBy).

1. The control-plane audit records route-table writes with operator, scope and address: platformctl routes audit, or GET /v1/audit on the control plane. An unexpected route change or rollback is a sign of a stolen config token.
2. The Data audit lists committed mutations per workspace with principal, request id and the policy that allowed them; it needs data.audit.read and is kept 90 days, so export the window you need before retention deletes it (Data API: read the audit, Data operations: retention). platformctl data audit and the console's audit page read the same trail.
3. The logs. Every service writes JSON lines carrying the request's x-request-id; search all three for it (gateway troubleshooting: start here).

The reason behind a gateway denial never reaches the caller and is only in Access's decision log line. No log line carries a token, secret, cookie or Authorization header.

1. The metrics show the shape and timing of an attack:

5. Report it#

If the incident reveals a vulnerability in the platform itself, report it privately, never in a public issue, as SECURITY.md describes: open a security advisory on the repository, or write to security@ggpwnkthx.com. Include the service, its version, steps to reproduce and the impact you expect. Each service reports its version on GET /healthz, and platformctl version shows the versions the deployment reports. The maintainers confirm or close the finding, prepare a fix, release it and then publish the advisory.

6. Restore normal service#

Once the secrets are replaced and the cause is understood, enable the tenant or principal again by re-sending it with "enabled": true or "status": "active", give back the grants that were legitimate, and remove any route overrides you no longer need. Review the Access hardening list: moving gateways to tenant credentials and unsetting ACCESS_DECISION_TOKEN removes the one credential whose reach is the whole platform.

Check that it worked#

• platformctl access decide --subject "$USER_ID" --workspace "$WORKSPACE" --route workspace-api exits 6, and Access's log gives the principal is disabled or the tenant is disabled as the reason.
• A new sign-in by the account ends on Access's page with 403 principal_disabled or 403 tenant_disabled.
• The session revocation and decision purge each reported a count, and the subject's next request through the gateway gets 401 or 403 workspace_forbidden rather than an upstream answer.
• After rotation, the old credential gets 401 and the new one works (see Rotate secrets).
• The audit export covers the incident window and is stored outside the affected deployment.

If something goes wrong#

Related#

• SECURITY.md
• Platform security
• Rotate secrets
• Manage access
• Troubleshoot a request
• Monitor the platform
• Back up and restore