Incidents
2 activeDeclared incidents, recurring failure fingerprints mined from failed attempts, and the circuit breakers currently shaping routing decisions.
Open & monitoring
Incidents requiring attention or under active watch.
P95 latency on gemini-3-pro up 2.1x since 06:00 UTC. Router latency penalty is steering interactive traffic to us-east-1 and ap-southeast-1 candidates.
node_kl_cafe_01 public IP changed twice in 40 minutes and failed re-attestation. Node moved to quarantine tier; account bindings failed over to KL office nodes.
Resolved
Closed incidents kept for the audit trail.
Sustained 429s on enterprise pool between 09:12 and 09:47 SGT; resilience engine shifted 34% of traffic to Bedrock provisioned throughput. No client-visible failures.
Failure fingerprints
Failed attempts grouped by provider × node × error class. Recurring fingerprints are what trip circuits and open incidents.
| Provider | Node | Error class | Count | First seen | Last seen | Circuit | Sample |
|---|---|---|---|---|---|---|---|
| anthropic | node_cf_use1_01 | provider_overloaded | 8 | 11d ago | 9h ago | — | gen_rchmbeke25 |
| anthropic | node_sg_dc_01 | rate_limited | 4 | 9d ago | 13h ago | half-open | gen_u1l3vrtgkk |
| openai | node_cf_use1_01 | network_timeout | 2 | 11d ago | 9h ago | — | gen_rchmbeke25 |
| anthropic | node_sg_dc_01 | egress_unavailable | 1 | 2d ago | 2d ago | half-open | gen_ffqgxgr837 |
| openai | node_cf_euw1_01 | credential_expired | 1 | 5d ago | 5d ago | — | gen_phlzl11ge9 |
Circuit breakers
Per-scope breakers consulted on every route plan. Open circuits remove their scope from candidacy until the cooldown elapses; half-open circuits admit probe traffic. Reset closes a breaker immediately; Open is the operator kill switch for a scope.
| Circuit | Level | State | Consec. failures | Last error | Opened | Cooldown | Actions |
|---|---|---|---|---|---|---|---|
| node:node_kl_cafe_01 | node | open | 17 | egress_unavailable | 2d ago | 86400.00 s | |
| account-node:acct_openai_pool_b_apse1@node_sg_dc_01 | account-node | half-open | 4 | rate_limited | 10m ago | 300.00 s | |
| provider:google-vertex | provider | closed | 1 | provider_overloaded | — | 120.00 s |