The Runbook Nobody Follows Because It's Already Stale

During an incident, the on-call engineer had the runbook open in one tab and was doing something completely different in the other, because step three of the runbook referenced a dashboard that had been replaced four months earlier. Nobody had deleted the runbook. Nobody was maliciously ignoring it. It had just quietly stopped being the thing anyone actually used, and the gap between "we have a runbook for this" and "this runbook works" had been invisible until an incident forced the question.

Runbooks decay faster than the systems they describe

A runbook is a snapshot of how to operate a system at the moment it was written, and systems change continuously — dashboards get redesigned, services get renamed, the person who used to get paged for a specific alert moves teams. None of those changes are runbook changes; nobody updating a dashboard thinks "I should also go check whether any runbook links to the old one." The runbook doesn't get updated when the system changes, because updating the runbook was never part of the process that changes the system. It only gets updated, if at all, the next time someone actually tries to use it and discovers it's wrong — which for most runbooks, covering low-frequency incidents, might be months or years between attempts.

That gap compounds. A runbook that's slightly stale after one undocumented change is still mostly usable. A runbook that's accumulated three or four small drifts since anyone last touched it stops being a reliable set of instructions and starts being a rough suggestion that an experienced engineer has to mentally patch in real time — which defeats the actual purpose of having a runbook, which is to let someone who isn't already deeply familiar with the system handle the incident correctly.

Testing a runbook by reading it tells you nothing

The natural review process for a runbook is reading it and checking that it makes sense, and that check will pass on a stale runbook just as easily as a correct one, because staleness shows up as a broken reference to something that no longer exists, and a document reads coherently right up until someone tries to actually click the link or run the command. The only check that actually catches drift is periodically executing the runbook for real, against the current system, during a scheduled low-stakes window rather than a live incident — a fire drill, not a document review.

That's a meaningfully more expensive check than a read-through, which is exactly why most teams do the cheap version and call it done. The cost asymmetry is real: a quarterly executed fire drill for every runbook is genuinely expensive in engineer time. But the alternative cost — discovering a runbook is wrong for the first time during a live incident it was supposed to shorten — is worse, and shows up at the worst possible moment, which is exactly what happened here.

What the incident actually cost

The specific outage this runbook was written for took roughly forty minutes longer to resolve than it should have, entirely spent on the engineer independently re-deriving the correct current dashboard and escalation path that the runbook was supposed to already contain. Forty minutes is a real, measurable cost that a periodic fire drill — an hour of scheduled, low-stakes practice per quarter — would have prevented outright, by catching the stale dashboard reference on a Tuesday afternoon instead of during an active incident.

A runbook that hasn't been executed recently isn't a slightly weaker version of a good runbook. Past a certain amount of undetected drift, it's closer to documentation of a system that used to exist, and the only way to know which one you have is to actually run it and see.