Technical Leadership & Mentoring • Practice Drill

Incident Triage: Rollback vs. Forward-Fix When Telemetry Is Incomplete

Hard Difficulty10 minYour Role: Engineering Team Lead / Incident Lead
Conversation Objective

Separate vanity HTTP 200 averages from p99 tail-latency and trace signals; favor rapid flag-based containment over unvetted hotfixes to protect DORA MTTR; keep triage blameless and schedule telemetry alerting improvements.

Workplace Scenario & Dynamics

You are Incident Lead on a triage bridge 25 minutes after release 4.18. Staff SRE Nadia Okafor wants 12 minutes to ship a 10-line forward-fix.

Release 4.18 went out 25 minutes ago. The primary API gateway HTTP 200 chart remains at 99.94%, yet distributed traces show downstream webhook p99 latency at 4x baseline and structured logs reveal 3% silent payload truncation on partial refunds. Rolling back the schema migration locks the refund table for 12 minutes, while a feature flag isolates the v2 serializer.

Realistic Pushback to Expect

Your counterpart will test your communication and composure. Expect reactions like:

“Primary checkout HTTP 200s are sitting at 99.94%; why tear down the release for a 10-line serializer edge case?”
“Rolling back the migration locks the refund table for 12 minutes; my hotfix takes 12 minutes anyway.”
“Finance is asking why our paging alerts did not catch silent truncation before support flagged it.”
What Good Looks Like in This Conversation
1. Diagnose FirstUnpack their mindset and constraints before prescribing fixes.
2. Clear BoundariesFrame standards with empathy without taking on their job.
3. Mutual OwnershipAgree on concrete next steps and check-in timeline.
Suggested Opening Angle

You can type your own opening or start with this suggested angle in the simulator:

"Nadia, I see the gateway 200s look green, and I want to compare that against the webhook p99 spike and refund trace truncation before we choose a fix."
Competencies:#technical-leadership#observability#dora-metrics#incident-response#sre#Calm#Direct#Analytical#Decisive
Conversation Partner
Nadia Okafor

Nadia Okafor

Persona

Fatigued Staff SRE pushing for a quick hotfix because HTTP 200s look healthy

Stance: defensive

Private Practice Instant Scorecard
Expensing through your team?Employer Pitch Template