The outage that wasn't: how a 2-second checkpoint stall masquerades as a DB crash

There's a specific kind of dread that hits when you're watching database connection errors stack up in your logs. Your stomach drops. You start mentally drafting the incident report. Then, thirty seconds later, everything is fine. No restart. No fix. No explanation.

That happened to me recently, and my first instinct was wrong.

What It Looked Like

The error burst lasted maybe twenty seconds. Connections timing out, a handful of refused connections, the application throwing fits. Then silence. Clean logs. Normal response times. Nothing changed on my end.

The pattern pointed straight at the database. So I went there first.

I checked the database process. Running. Checked active connections. Normal. Checked query times in the period around the incident. Nothing alarming. The database had no idea anything happened.

That's the first clue, and I almost missed it.

The Diagnostic Fork in the Road

Here's the thing about a real database outage: it doesn't fix itself. If the process crashes, something has to restart it. If you hit connection pool exhaustion, connections stay exhausted until they're released or you intervene. If there's a lock storm or a runaway query, it leaves a mark in the slow query log.

A true outage has evidence. It has a before and after. It requires a human or an automated recovery process to close the gap.

What I had instead:

Duration: Roughly 20 seconds, then fully resolved
Frequency: Not the first time (I found two similar blips looking back through logs)
Database-side evidence: None
Recovery action required: None

That pattern doesn't describe an outage. It describes a stall.

What Was Actually Happening

I pulled infra-level metrics. Not application logs. Not database logs. The actual disk I/O metrics for the host the database was running on.

There it was. A brief, sharp spike in I/O wait time that lined up exactly with the error window. CPU was flat. Memory was fine. Connection counts were normal before, during, and after. Just the disk.

What I was seeing was a checkpoint stall. The database's storage engine periodically flushes dirty pages from memory to disk. This is routine. It's expected. Under normal conditions it happens in the background and you never notice. But when that flush takes longer than usual, the database engine briefly blocks new write operations while it catches up. From the application's perspective, this is indistinguishable from the database being down. Connections time out. Requests fail. Then the flush completes, writes unblock, and everything recovers on its own.

The database wasn't down. The disk was briefly busy, and the database was waiting on it.

Why I Lost Time

I spent the first several minutes of that incident looking in the wrong place. I was in database logs, checking process status, reviewing connection pool settings. All reasonable instincts. All pointed at the wrong layer.

The mistake wasn't being wrong. The mistake was treating "the application sees database errors" as equivalent to "the database has a problem." Those are not the same statement.

Application-layer errors tell you what the app experienced. They don't tell you why. A checkpoint stall and a crashed database process produce identical symptoms at the application layer. The only way to tell them apart is to go one layer deeper and look at what the disk was doing.

A transient error burst that self-resolves is a fundamentally different debugging problem than a sustained one. If errors are still happening when you're investigating, you have time to chase the cause. If they've already cleared, you're doing forensics. And forensics requires infrastructure metrics, not just application logs.

What I Do Now

When I see a self-clearing error burst, I check three things before I do anything else:

Did the database show any internal evidence? Slow query log, error log, crash/restart markers.
Did connection pool metrics spike? Exhaustion leaves a trace.
What was disk I/O doing? Specifically I/O wait, not just throughput.

If the first two are clean and the third shows a spike, it's almost certainly a checkpoint stall or a similar flush event. Not an outage. Just a background maintenance process doing its job louder than usual.

The fix, if you need one, lives at the storage configuration level. That's a separate conversation. But you can't get there if you spend the incident convinced the database is the problem.

The error message said "database connection failed." The database was fine. The disk needed a moment. They just didn't bother to cc me on that.