Troubleshoot a request
Follow one failing request across the platform: read what came back, work out which service produced it, trace it through the logs by its request ID, and jump to the troubleshooting entry that fixes it. Developers start here when a call fails (developer); operators start here when a user reports a request ID (sysops).
Read byPlatform operatorApplication developerConsole and CLI user
Before you start#
| You need | Why |
|---|---|
The full response: status line, headers and body (curl -i, the browser's network panel, or the console's refusal panel) | The status alone is ambiguous: several services use the same numbers |
| Read access to the gateway, Access and Data logs | Each service logs the reason it never sends to the caller |
| The metrics token for each service, if you want counters | /metrics answers 404 or 401 without it |
platformctl with a profile for the deployment (optional) | status, doctor and access decide shorten the search (CLI usage) |
Steps#
1. Capture the answer and its request ID#
Every gateway answer carries x-request-id, and the same ID is on every gateway log line of that request. Copy it together with the status, the body and any retry-after header.
You can choose the ID yourself when you reproduce the problem: the gateway keeps a caller's x-request-id when it matches ^[A-Za-z0-9._-]{1,64}$ and replaces anything else with a UUID (gateway spec §2.1). The CLI sends one with --request-id:
curl -s -i -H 'x-request-id: repro-001' "https://api.example.com/ws/<workspace>/v1/me" -H "cookie: $SESSION"
platformctl --request-id repro-002 --verbose data meWhen the console refuses something, its page shows the status, the code, the message and the request ID (console user guide).
2. Find which service answered#
Read the body before the status. Each service has its own error shape, and the gateway forwards an upstream's body untouched (gateway spec §2.3):
| Body | Answered by | Codes are listed in |
|---|---|---|
{"error": "no_route", "message": "…"}: a string error, lower-case code | The gateway itself | gateway spec §2.5 |
{"error": {"code": "FORBIDDEN", "message": "…"}, "requestId": "…"}: an object error, upper-case code | Data, forwarded by the gateway | data spec §9.7 |
Empty, with status 499 | Nobody: the caller left before the headers | gateway spec §11.3 |
| Any other shape | Another upstream, forwarded unchanged | That service's own docs |
| A login page with a status and a code, on Access's address | Access, during sign-in | Access troubleshooting: failed logins |
{"error": "<code>", "message": "…"} from Access's own address | Access administration or a contract endpoint, called directly | Access troubleshooting |
| A message on standard error and an exit status | The CLI, reporting a service's refusal | CLI spec §6.2 |
An Access decision never reaches the caller as such: the gateway turns it into its own 403 workspace_forbidden or 503 access_unavailable and keeps the reason out of the answer (gateway spec §6).
3. Trace the request through the logs#
- Search the gateway logs for the request ID. The line names the route, the caller's subject, the workspace and the trace ID (gateway spec §12.2).
- The gateway sends the upstream a
traceparentwith that trace ID, so an upstream that logs trace context can be searched with it (gateway spec §7.5). - For Data, look up the gateway's request ID in the Data log (Data troubleshooting: gateway answers 502). A Data error body also carries
requestId, the identifier Data's own log lines use (data spec §15.2). - For a
403 workspace_forbidden, read Access's decision log line for the same subject, workspace and route: it holds the reason and its code (Access troubleshooting: decisions denied).
If the gateway logs have no line for the ID, the request never reached the gateway, or it reached it with a malformed ID that was replaced: check the ingress first.
4. Check the services' health#
Rule out an outage before you debug one request:
platformctl status # health and readiness of the services in the profile; --watch repeats every 5 s
platformctl doctor # also checks the profile and your credentialBoth end with exit status 7 when a service does not answer (CLI spec §5.1). Without the CLI, read each service's /readyz body: the gateway reports its route table and session store (gateway spec §12.3), and Data gives a reason (data spec §15.1). The console's Status page shows the same probes (console user guide).
5. Ask Access the gateway's question#
For any refusal that involves a caller's rights, ask Access directly with the subject, workspace and route from the gateway's log line. A denial ends with exit status 6:
platformctl access decide --subject <principal id> --workspace <workspace uuid> --route <route id>6. Look the answer up#
Find the status and code in this table, then follow the link. Codes in lower case come from the gateway, codes in upper case from Data.
| Status and code | What it usually means | Go to |
|---|---|---|
302 to /auth/login | A browser without a valid session on a route that accepts one | gateway spec §5.2 |
400 bad_request | The workspace placeholder is missing or not a UUID | Gateway troubleshooting |
400 login_failed | The login callback could not complete | Gateway troubleshooting |
400 INVALID_QUERY, INVALID_REQUEST, INVALID_CURSOR, INVALID_PRECONDITION | Data refused the query, body, cursor or if-match | data spec §9.3, §9.4, §9.5 |
401 unauthorized | No valid credential for the route's mode | Gateway troubleshooting |
401 UNAUTHENTICATED | Data could not verify the gateway's signature, or the call bypassed the gateway | Data troubleshooting |
403 workspace_forbidden | Access denied the caller the workspace on this route | Gateway troubleshooting, Access: decisions denied |
403 cross_site_request | A request with the session cookie came from a site that is not allowed | Gateway troubleshooting |
403 FORBIDDEN | The gateway let the caller in, but Data's context or policies refuse the action | Data troubleshooting |
403 identity_unknown and other login-page codes | Access refused the sign-in | Access troubleshooting |
404 no_route | No route matches the host, path and method | Gateway troubleshooting |
404 not_found | The path is reserved by the gateway | Gateway troubleshooting |
404 NOT_FOUND | Unknown or disabled workspace, a collection the workspace does not serve, or a record out of reach | Data troubleshooting |
409 CONFLICT, IDEMPOTENCY_CONFLICT, OPERATION_IN_PROGRESS, CURSOR_STALE | A concurrent write, a reused key, a create still running, or an outdated cursor | Data troubleshooting |
409 from the control plane | Stale If-Match, or a protected route | Gateway troubleshooting |
412 PRECONDITION_FAILED | The record changed after you read its etag | Data troubleshooting |
413 payload_too_large | The body is over the route's limit | Gateway troubleshooting |
422 VALIDATION_FAILED, INVALID_REFERENCE | The record does not fit the manifest, or a reference is not readable | data spec §9.7 |
422 RESULT_TOO_LARGE | The page or its expansions passed Data's budget | Data troubleshooting |
422 from the control plane | The route table is invalid | Gateway troubleshooting |
428 PRECONDITION_REQUIRED | An update, delete or rollback without if-match | data spec §9.4 |
429 too_many_requests | A gateway rate limit; wait for retry-after | Gateway troubleshooting |
429 TOO_MANY_REQUESTS | The tenant used its share of Data's concurrent requests | Data troubleshooting |
500 internal_error | The subject cannot be written to a header | Gateway troubleshooting |
500 INTERNAL_ERROR | A Data defect; report it with the request ID | data spec §9.7 |
502 bad_gateway | The upstream was unreachable, refused TLS, or broke the exchange | Gateway troubleshooting, Data troubleshooting |
503 not_ready | The gateway never loaded a route table | Gateway troubleshooting |
503 gateway_overloaded | Admission or the exchange cap is full, usually behind a slow dependency | Gateway troubleshooting |
503 access_unavailable | The gateway could not get a decision from Access | Gateway troubleshooting, Access: outages |
503 identity_provider_unavailable, session_store_unavailable | Sign-in cannot reach the provider or store the session | Gateway troubleshooting |
503 AUTHORIZATION_UNAVAILABLE | Data could not get a usable access context from Access | Data troubleshooting |
503 DATABASE_UNAVAILABLE | Data cannot reach its database or registry | Data troubleshooting |
503 OVERLOADED, DRAINING | The Data process is full, or shutting down | Data configuration: budgets, Data operations: drain |
504 gateway_timeout | The upstream sent no headers within the route's responseMs | Gateway troubleshooting |
504 DEADLINE_EXCEEDED | A Data statement ran over its budget | Data troubleshooting |
Two pairs are easy to confuse. A 403 workspace_forbidden comes before the request reaches Data: Access found no live grant for that workspace and route, or the principal or tenant is disabled. A 403 FORBIDDEN comes from Data, often for a caller who does hold a grant whose permission keys no policy maps to the requested action (data spec §2.3). And with default settings, the gateway gives up on an upstream after 5 seconds while Data allows a request 10 seconds (DATA_REQUEST_TIMEOUT_MS), so a slow Data call usually surfaces as the gateway's 504 gateway_timeout.
Check that it worked#
- Send the same request again with a fresh
x-request-idand confirm the status you expect. - Find that ID in the gateway log and, for Data routes, in the Data log, with no refusal line.
- Watch the counter that matched the symptom stop rising, for example
gateway_auth_failures_total,gateway_access_unavailable_totalordata_authorization_denials_total(Gateway troubleshooting: start here, Data operations: metrics). platformctl doctorends with exit status0.
If something goes wrong#
| Symptom | Look here |
|---|---|
| No gateway log line carries the request ID | Gateway operations: ingress |
| The status is not in the table above | gateway spec §2.5, data spec §9.7, Access troubleshooting |
platformctl refuses before sending anything | CLI usage: when something refuses |
A WebSocket closes with 1008 or 1001 | Gateway troubleshooting: connections close |
| Every caller fails at once | Monitor the platform, Respond to an incident |