Rollback Runbook
Problem
Section titled “Problem”A deploy is causing errors, elevated latency, or otherwise needs to be undone in production, now.
Severity
Section titled “Severity”Typically SEV1/SEV2 — you’re already mid-incident if you’re here.
Symptoms
Section titled “Symptoms”Error-rate spike, latency spike, or a specific broken feature that started right after a deploy.
Detection
Section titled “Detection”Correlate the start time of the symptom with the last deploy timestamp. If they line up, this is a rollback, not a root-cause hunt — investigate after service is restored.
Immediate Checks
Section titled “Immediate Checks”Confirm this is deploy-correlated
If the last deploy was more than ~30 minutes before symptom onset, this is probably not a rollback situation — stop and diagnose instead.
Mitigation
Section titled “Mitigation”Cut nginx back to the previous known-good instance
Reverse the cutover from the last deploy — point cutover-live.conf back
at the prior healthy instance.
Confirm traffic is flowing to the old instance
Check the API smoke test (GET /api/v1/catalog) and error-rate telemetry.
Announce the rollback
Post in the incident channel with what was rolled back and why.
Recovery
Section titled “Recovery”Confirm error rate and latency return to baseline within a few minutes of cutover. If they don’t, the deploy wasn’t the cause — escalate to broader diagnosis.
Validation
Section titled “Validation”Watch telemetry for at least 15 minutes post-rollback before declaring the incident mitigated.
Escalation
Section titled “Escalation”If rollback doesn’t resolve symptoms, this isn’t a simple deploy issue — page the on-call platform engineer for broader diagnosis.
Follow-Up
Section titled “Follow-Up”Every rollback gets an incident doc and, if customer-impacting, a postmortem. See INC-0001 for the shape.