Debugging an Incident That Only Reproduces Under Load

The bug report said the checkout service occasionally returned stale inventory counts. Every attempt to reproduce it locally, with a single request against a single instance, succeeded cleanly every time. The bug was real, it was hitting real customers, and it was completely invisible to any debugging process built around running one request and watching what happens.

Load isn't an amplifier, it's a different code path

The instinct when something "only happens under load" is to treat load as a stress multiplier — the same bug, just happening more often because there are more chances for it to trigger. Sometimes that's true. Here it wasn't. Under load, requests were hitting the inventory cache concurrently in a window where one request's read and another request's write to the same key interleaved in a specific order that single-request testing could never produce, because single-request testing has no second request to interleave with. This wasn't the same bug happening more often. It was a bug that only exists as a possible code path when concurrency is present at all — a race condition doesn't have a "mild" version that shows up at low traffic, it either fires or it can't fire, and it can't fire with one request in flight.

Debugging this by making single requests and reading logs was never going to work, not because I wasn't looking hard enough, but because the failure mode I was looking for structurally cannot occur in the environment I was testing it in.

Reproducing it meant reproducing the actual condition, not the symptom

The fix wasn't more careful manual testing. It was writing a load test that fired concurrent requests specifically targeting the same inventory key, because the symptom — occasional stale reads — was a description of the outcome, and the outcome only tells you where to look, not what condition produces it. Once concurrent requests against a shared key were actually happening, the race reproduced within seconds, consistently, because the condition that had been rare in production traffic (two requests for the same popular item landing close enough together to interleave) was now happening on every test run by construction.

That's the general shape of load-dependent bugs: the report describes an outcome, and the outcome is compatible with several possible causes, but only one of them requires concurrency to exist as a code path at all. Ruling those in or out means asking, before writing any reproduction, whether the suspected mechanism is even possible without concurrent execution. If it isn't, no amount of careful single-request testing will ever find it, no matter how many times you run it.

What actually caught it in the log, after the fact

Once the race was understood, going back to production logs showed the signature clearly — two requests for the same inventory key with overlapping timestamps down to the millisecond, right before every stale-read incident on record. It had been there the whole time, visible in retrospect, invisible in advance, because nobody was looking for concurrent-request pairs specifically; the logs were being read one request at a time, the same way the local reproduction attempts had been run.

The lesson that generalizes: an incident that resists single-request reproduction is telling you something about its own mechanism before you've even found the mechanism — it's telling you the mechanism probably requires more than one request in flight. That's a testable hypothesis on its own, well before you know what the race actually is, and it changes what kind of reproduction is worth attempting next.