Monitor the platform

Before you start#

Steps#

1. Wire the probes into the orchestrator#

Use GET /healthz for liveness and GET /readyz for readiness and rollout gating on every process. /healthz never calls a dependency, so a failing database does not get a healthy pod killed; /readyz takes a replica out of rotation while a dependency it needs is down or while it drains (platform probes).

A failed route-table reload does not make the gateway unready: the last valid table keeps serving and the error shows in the /readyz body as config.lastError (gateway spec §12.3). Data's 503 body carries a reason naming the failing part (Data spec §15.1).

Set each pod's termination grace period above its drain budget so the drain finishes before the kill. From an operator machine, platformctl status --watch polls the same probes every five seconds (CLI usage), and the console's Status page shows them to anyone (console user guide).

2. Give the scraper its tokens#

Each metrics endpoint is guarded by its own bearer token and exists only once you configure that token. Send the token as Authorization: Bearer <token> and keep the endpoints off the public ingress.

curl -sS -H "Authorization: Bearer $GATEWAY_METRICS_TOKEN" http://gateway:8080/metrics | head
curl -sS -H "Authorization: Bearer $ACCESS_METRICS_TOKEN"  http://access:8081/metrics | head
curl -sS -H "Authorization: Bearer $(cat /run/secrets/data-metrics-token)" http://data:8082/metrics | head

The console and the control plane document no metrics endpoint; watch them through their probes and logs. Every label set is bounded on purpose (gateway routes cap at 500, Access tenants at 100, and Data carries no tenant, principal or workspace label at all), so per-tenant dashboards come from logs, not from metrics.

3. Alert on the gateway#

The gateway sits on every request, so its metrics show most incidents first. Metric names and labels are in gateway spec §12.1; the alert list is in gateway monitoring.

For latency, gateway_request_duration_seconds, gateway_upstream_duration_seconds and gateway_access_duration_seconds split the time between the gateway, the upstream and Access. gateway_auth_failures_total{reason} separates missing_role from access_denied.

4. Alert on Access#

Access documents eight alerts with expressions, durations and runbooks in Access monitoring. Load them as written; the expressions look like this:

rate(access_decisions_total{result="deny"}[5m]) / rate(access_decisions_total[5m]) > 0.2      for 10m
histogram_quantile(0.99, rate(access_decision_duration_seconds_bucket[5m])) > 0.02             for 5m
rate(access_logins_total{result!="success"}[10m]) > 0.1                                         for 10m
rate(access_store_errors_total[5m]) > 0                                                         for 5m
pg_stat_activity_count{application_name='access'} > 7                                           for 5m
rate(access_admin_requests_total{status="401"}[5m]) > 0                                         for 5m
probe_success{job="access"} == 0                                                                for 2m

The eighth, a schema older than the code, is not a metric: new replicas stay 503 not-ready and the start log names an unknown migration, so alert on readiness plus that log line. The page also lists four platform-level signals to add: replicas not ready, pod memory against 200 MB, the ingress certificate, and the age of the last database dump.

Decision reasons never reach a metric or the caller; they are in Access's decision log line (Access spec §12.3).

5. Alert on Data#

Data's metric names are in Data metrics and its alarms in Data alarms.

Worker processes become unready without a delivery store or a registered subscriber, so probe them as well as the API.

6. Watch the console and the control plane#

The console logs one JSON line per request but skips probe requests, and its readiness follows Access's discovery document (console probes and logs). A console restart signs everyone out, so count restarts.

The control plane answers 503 to every request while the shared state is unreachable, and five failed authentications in sixty seconds from one address earn 429. Every write it accepts is in GET /v1/audit, or platformctl routes audit; review it after route-table changes (gateway spec §12.7).

7. Collect logs and follow one request ID#

Every service writes JSON to stdout with secrets redacted, and every service echoes an x-request-id:

To trace a failing request, take the x-request-id from the response (or the console's error page, or set your own with platformctl --request-id), search the gateway logs for it, then search Access's decision line and Data's line for the same ID. The gateway's running-policy lines, logged once per route-table generation, also flag permissive routes with warnings such as unauthenticated_route (gateway spec §12.2). Never rename a Data log field that carries a secret: redaction matches field names (Data logs). A full walk-through is in Troubleshoot a request.

8. Decide what pages a person#

The service pages agree on these conditions as worth an alert; route each to someone who can follow the runbook.

Check that it worked#

1. Each curl in step 2 returns Prometheus text; the same call without the header returns 401 from the gateway and Data.
2. Your scraper's target list shows every gateway, Access and Data replica as up.
3. In staging, send SIGTERM to one replica of each service: its /readyz turns 503 with a draining body, the orchestrator stops routing to it, and your readiness alert fires if the replica does not come back.
4. Send a request through the gateway with x-request-id: monitor-check-1, then find that ID in the gateway, Access and Data logs.
5. Open the console's Status page with console.status and confirm versions and latencies for the three services.

If something goes wrong#

Related#

• Platform operations: probes
• Gateway monitoring
• Access monitoring
• Data alarms
• Console probes and logs
• Troubleshoot a request
• Respond to an incident
• Rotate secrets