INC-0001: Catalog endpoint latency spike
Timeline
Section titled “Timeline”All times UTC.
| Time | Event |
|---|---|
| 14:02 | GET /api/v1/catalog P99 latency alert fires |
| 14:05 | On-call platform engineer acknowledges |
| 14:11 | Correlated with a deploy at 13:55; rollback initiated |
| 14:16 | Latency returns to baseline |
| 14:30 | Incident closed |
Impact
Section titled “Impact”Elevated latency (P99 ~4s vs. ~200ms baseline) on the catalog endpoint for ~14 minutes. No errors, no data loss; Tidepool catalog refreshes were slow but not failing.
Detection
Section titled “Detection”Automated P99 latency alert (see Patch API for the service this alert covers).
Response
Section titled “Response”On-call platform engineer followed the Rollback Runbook after confirming deploy correlation.
Root Cause
Section titled “Root Cause”Example: a new catalog query in the 13:55 deploy was missing an index used by the previous query shape.
Contributing Factors
Section titled “Contributing Factors”Example: the query change wasn’t covered by a load test that would have caught the missing index before production.
Resolution
Section titled “Resolution”Rollback to the prior deploy resolved the symptom immediately.
Corrective Actions
Section titled “Corrective Actions”| Action | Owner | Due | Status |
|---|---|---|---|
| Add the missing index | platform | — | example |
| Add a load test for catalog query shape changes | platform | — | example |
See postmortem.