An API That Returns 200 for a Request It Silently Dropped

A webhook consumer had a 100% success rate on every request it received, according to its own logs. It also wasn't processing about six percent of the events it was supposed to. Both facts were true, and reconciling them meant realizing the success rate was measuring whether a response got sent, not whether anything the response implied had actually happened.

A status code is a claim, not a receipt

The endpoint accepted a webhook payload, enqueued it for background processing, and immediately returned 200 — a completely standard pattern for keeping a webhook sender from timing out while real processing happens asynchronously. The gap was that the enqueue step could fail silently past a certain queue depth, dropping the payload, while the 200 had already been decided before the enqueue call even ran. The response code was answering "did I receive and acknowledge this request," which was always yes. It was not answering "will this event actually be processed," which was sometimes no, and nothing in the response distinguished those two very different claims.

This is a common enough pattern that it's worth naming precisely: any handler that returns success before its real work is guaranteed to complete is making a promise it can't actually back. That's not automatically wrong — plenty of systems need to ack fast and process later — but it means the response code has stopped being a reliable signal for the thing people assume it signals, and every consumer of that API needs to know that, explicitly, rather than assuming REST convention holds.

Why nobody caught it for months

The webhook sender's own dashboard showed a clean delivery rate, because delivery, from its point of view, meant getting a 200 back — which it always got. The receiving service's logs showed no errors, because nothing on the receiving side ever threw; the silent drop happened in a code path that failed by returning early, not by raising. Every system involved was reporting accurately on the narrow thing it was actually checking, and none of those narrow things happened to be "did the event get processed," which was the only question anyone actually cared about.

Finding the gap required a check that didn't exist yet: counting events sent against events actually reflected in the downstream data they were supposed to update, over a matching time window. That comparison — sent versus effected, not sent versus acknowledged — was the only place the six percent gap was visible, because it was the only check that looked past the response code to the actual outcome the response code was implicitly promising.

The fix, and the harder lesson under it

The immediate fix was mechanical: make the enqueue step's failure visible by returning an error status when it fails, instead of ack-then-drop. The harder lesson is that "the API returned success" and "the thing the API call was for actually happened" are two different claims that happen to usually coincide, closely enough that most systems never bother distinguishing them — until a failure mode shows up specifically in the gap between them, in which case every dashboard built around response codes turns out to have been measuring the wrong thing the entire time.

Any system that acks before completing real work needs a downstream check that verifies the completion independently of the ack. Otherwise the only failures visible are the ones that fail loudly enough to break the ack itself — and the quiet ones, the ones that succeed at responding and fail at doing, run invisibly for as long as nobody thinks to check the actual effect against the actual intent.