Incident Triage: Rollback vs. Forward-Fix When Telemetry Is Incomplete
Separate vanity HTTP 200 averages from p99 tail-latency and trace signals; favor rapid flag-based containment over unvetted hotfixes to protect DORA MTTR; keep triage blameless and schedule telemetry alerting improvements.
You are Incident Lead on a triage bridge 25 minutes after release 4.18. Staff SRE Nadia Okafor wants 12 minutes to ship a 10-line forward-fix.
Release 4.18 went out 25 minutes ago. The primary API gateway HTTP 200 chart remains at 99.94%, yet distributed traces show downstream webhook p99 latency at 4x baseline and structured logs reveal 3% silent payload truncation on partial refunds. Rolling back the schema migration locks the refund table for 12 minutes, while a feature flag isolates the v2 serializer.
Your counterpart will test your communication and composure. Expect reactions like:
You can type your own opening or start with this suggested angle in the simulator:
"Nadia, I see the gateway 200s look green, and I want to compare that against the webhook p99 spike and refund trace truncation before we choose a fix."
Nadia Okafor
PersonaFatigued Staff SRE pushing for a quick hotfix because HTTP 200s look healthy
Stance: defensive