Postmortem: Catalog endpoint latency spike
Blameless. This document explains what happened and what we’re changing — it does not assign fault to a person.
Summary
Section titled “Summary”A schema-mismatched query shipped in a routine deploy caused catalog lookups to run without an index for ~14 minutes before rollback.
Customer Impact
Section titled “Customer Impact”Slow catalog refreshes in Tidepool and the web app; no failed requests, no data loss.
Internal Impact
Section titled “Internal Impact”None beyond the response time of the on-call engineer.
Timeline
Section titled “Timeline”See INC-0001 — not duplicated here.
Root Cause
Section titled “Root Cause”A new query shape in the catalog handler wasn’t covered by the index the old query relied on implicitly.
What Worked
Section titled “What Worked”The latency alert fired within 2 minutes of the regression, and the Rollback Runbook resolved it in under 15.
What Failed
Section titled “What Failed”Nothing in CI caught a query-shape change that would degrade performance under a missing index — this class of regression isn’t covered by existing tests.
Corrective Actions
Section titled “Corrective Actions”| Action | Owner | Due | Status |
|---|---|---|---|
| Add the missing index | platform | — | example |
| Add a load test for catalog query shape changes | platform | — | example |
| Add a query-plan check to CI for the catalog handler | platform | — | example |
Owners
Section titled “Owners”Platform team owns follow-through on the corrective actions above.