Silent Webhook Failure Patterns and Retry Architecture

A webhook failure is dangerous precisely because nothing on the receiving end looks broken. The dashboard is green, and yet the record that was supposed to be written never lands in the database, or lands twice, or lands in a form the provider never sent. What the sender actually experienced, whether it considers that delivery a success or a failure, is invisible unless someone goes looking for it.

This is a property of the push-based, fire-and-forget model that webhooks are built on, not a defect specific to any one payment processor, CRM, or messaging platform. The sender initiates delivery and moves on; the receiver reacts whenever the request arrives, with no shared transaction spanning both sides. That asymmetry creates a gap between what the sender believes happened and what the receiver can actually confirm, and in that gap, data disappears without tripping a single alert.

The consequences are not abstract. A duplicate delivery can apply a billing credit twice. A subscription-deletion event that fires out of order, or twice, can revoke a paying customer's access. None of these outcomes requires a crash. They require only that the system behave exactly as instructed, following instructions that have already gone stale or duplicated.

A similar structural blind spot appears in multi-agent AI systems, where production reliability research describes a "false prosperity" pattern: external metrics stay normal for extended periods even as output quality has already degraded, because the agents keep accepting tasks and producing outputs that look complete. Webhooks and agent pipelines fail for different reasons, but they share the same underlying fault: the monitoring built around them measures whether something happened, not whether what happened was correct. That gap between what monitoring measures and what the system actually needs to be judged on is what makes both failure classes so hard to catch in production, and it is why the rest of this piece treats webhook observability as an architectural requirement.

The five failure modes

Webhook reliability is often treated as a single problem with a single fix: add retries, move on. In practice, five distinct failure modes recur across providers, and each one exploits a different blind spot in a receiver's observability setup. A defense built for one does little to catch the others.

Duplicate delivery is the first. Providers guarantee at-least-once delivery, not exactly-once, so even an endpoint that returns 200 on first receipt can see the same event arrive again. What makes duplicate delivery silent is that both the original and the repeat event process successfully. Nothing errors. The handler runs its business logic twice, and nothing in a standard stack is watching for that repetition on its own. A subtler version of the same problem is a race condition: if two duplicates arrive milliseconds apart, both can pass an existence check before either one finishes inserting its record, so idempotency has to be enforced inside a transaction with a unique constraint on the event ID, not just a prior lookup.

Signature verification failure is the second mode, and it traces back to how providers construct the signature. Providers compute the webhook signature against the exact raw bytes of the request body. If any middleware parses, transforms, or re-serializes that body before verification runs, the computed signature will not match what the provider sent. Express-style body parsers are a common trap, as is framework-level JSON parsing in Next.js routes, and so is base64 encoding applied by serverless platforms like Lambda or Vercel functions. What makes this mode silent is that the failure is loud in one narrow sense (verification fails, the event gets rejected, the sender may retry) and completely opaque in every other sense: the receiver's logs show a failed HMAC check with no indication of why it failed. The fix is to verify against the raw body string, typically something like req.text(), never against the already-parsed object.

DNS-level ghost endpoints form the third mode, and they hide differently from the first two because nothing ever reaches the receiver. Before a provider can deliver an event, it has to resolve the destination domain to an IP address. If that lookup fails, the request never leaves the provider's network, and not a single byte arrives at the receiver. Common causes include an endpoint URL still pointing at localhost or a private IP, a typo in the domain, or a stale DNS cache left over after a server migration. Application logs in this mode are completely empty, and that emptiness is itself the diagnostic clue. Debugging starts on the provider's side: checking the provider's own delivery logs for connection errors, confirming public DNS resolution independently, and temporarily pointing the endpoint at a public request-catcher to verify the provider can reach the open internet.

Handler crashes returning 500s make up the fourth mode, and what happens after the 500 depends entirely on which provider is sending the event. Stripe retries in live mode for up to three days with exponential backoff. Shopify, in a policy updated in September 2024, retries multiple times over a limited window with exponential backoff. GitHub does not automatically redeliver failed webhook deliveries at all: a server error means the event is gone unless someone catches it in the delivery history and redoes it manually. The fix is to log the raw body before any parsing happens, wrap the handler in a try/catch, return a 2xx response regardless, log the trace internally, and route the failed payload to a dead-letter queue rather than letting a bad payload trigger a retry storm.

The fifth mode is the false-200 trap, and it defeats the mechanisms built to catch failure more completely than the other four. A webhook endpoint acknowledges receipt with a 200 OK before the payload has actually been persisted. The handler returns success, then fails during the subsequent database write. From the sender's perspective, delivery succeeded, so every retry mechanism stands down. This mode is also the most common source of timeout-driven failures: doing heavy work such as database writes, external API calls, or email sends synchronously, before returning 200, can push the response past the provider's allowed window, causing the provider to treat the connection as failed and retry, even though the first attempt may have partially completed and left the system in an inconsistent state. Those response windows are tighter than most developers expect and vary sharply by provider: Slack's Events API enforces a strict few-second cutoff, Shopify enforces a short connection timeout inside a tight total window, GitHub enforces a strict 10-second timeout, and Stripe does not publish a fixed number but instructs handlers to return 2xx quickly, before any complex logic runs.

Why the false-200 trap demands a queue-first architecture

The false-200 trap cannot be patched at the handler level, because the handler is the thing lying to the sender. Fixing it requires separating the act of receiving an event from the act of processing it at the architectural level.

The canonical pattern looks like this: verify the signature against the raw payload, immediately place that raw payload into a message queue or a high-speed database, return 200 OK, and let a separate background worker pull the message and process it on its own schedule. The 200 response now certifies something true and durable (the event has been safely stored) rather than something optimistic and fragile (the event has been fully processed, including every downstream database write, API call, and email send, all within a few seconds).

That separation does more than close the timeout gap. It addresses all five failure modes described above, not just the slow-handler case that first motivates it. Duplicate delivery becomes a problem the queue is well-positioned to solve, since checking the event ID before enqueuing, or deduplicating before the worker ever touches the message, keeps a repeated delivery from running business logic twice. Handler crashes stop being catastrophic, because a worker that crashes mid-processing can be retried without the sender ever knowing or caring. The false-200 trap itself is removed by construction: the success response is returned after the event is safely enqueued, not after processing finishes, so the sender's record of success is now accurate. But a queue still reduces how damaging the other four modes become when they compound with a DNS problem, because durable storage limits how much gets lost once connectivity returns.

Some early-stage teams avoid this pattern because it adds infrastructure, namely a queue, where a direct synchronous handler had seemed sufficient. It says nothing about how long a failed delivery should wait before trying again, or when a provider's own retry behavior should override a computed delay. That is a retry policy question, and it needs its own design.

Building a retry policy that classifies failures before scheduling them

A retry policy that treats every failure identically, retrying everything on the same schedule or retrying nothing at all, either floods a struggling downstream dependency with unnecessary load or abandons events that were fully recoverable. Effective retry starts by classifying the failure before scheduling anything.

Some HTTP status codes are worth retrying and others are not. Retriable codes include 408 (Request Timeout), 429 (Too Many Requests), 500 (Internal Server Error), 502 (Bad Gateway), 503 (Service Unavailable), 504 (Gateway Timeout), and DNS resolution failures. Most 4xx client errors outside of 408 and 429 are not retriable: a malformed-payload error means the payload is malformed, an authentication error means authentication failed, and sending the identical request again changes nothing about either outcome. When a Retry-After header is present in the response, it should override whatever backoff interval a retry policy would otherwise compute, because the destination server knows its own recovery window better than an exponential curve guessed in advance. Retrying sooner than that header specifies just earns another 429.

A workable retry schedule splits into three tiers. The first tier handles transient network issues with two to three immediate attempts, spaced 100 to 500 milliseconds apart. The second tier handles temporary outages with exponential backoff and jitter spread across a window of a few hours, stepping through delays like one minute, five minutes, fifteen minutes, thirty minutes, and one hour. Stripe's own retry window runs three days, and that figure is a reasonable real-world ceiling to anchor a custom implementation against, since common implementations cap out somewhere between several days and a fixed number of attempts.

Jitter is not a refinement to add later. Without it, a burst of failures that all begin retrying at the same moment will re-arrive at the same moment again at every backoff interval, simply replacing one traffic spike with another, evenly spaced one. Randomizing the retry window spreads that load across time.

Each delivery attempt needs its own record: a timestamp, the delay that preceded it, the HTTP status or error returned, and the final disposition, whether that is delivered, still retrying, or dead-lettered. Once retries run out, though, tracking has to answer one more question: where does the event go, and can anyone actually do something with it there.

Dead-letter queues and the observability layer that makes retry trustworthy

A retry policy that cannot be observed cannot be trusted, and a failure that exhausts every retry attempt without a durable landing place is simply lost. That outcome is functionally identical to never having had a retry policy.

When a webhook exhausts its retry schedule, the payload should move to a dead-letter queue, a dedicated table or queue holding permanently failed deliveries so a person can inspect what went wrong. For that inspection to be useful, the DLQ has to preserve the original payload, the retry history, and the reason the delivery failed. The dead-letter queue is not where failed events go to be forgotten. It exists so that once the receiver's bug is patched, the event can be replayed and the data it was carrying can still land correctly.

Observability around retry has to cover both the individual attempt and the pattern across attempts. At the level of a single delivery, that means the attempt timestamp, the delay that preceded it, the outcome, and the final disposition. At the aggregate level, retry rate over time functions as an early-warning signal, while dead-letter queue depth functions as a lagging indicator that a receiver problem has been running longer than the retry window allows. Even silence carries information: if an endpoint that normally receives a steady stream of events suddenly shows no log entries at all, that absence may point back to a DNS ghost endpoint, the third failure mode, and an alert tuned to expected events per interval is what catches a gap like that.

Research on evaluation blindness in AI systems gives this problem a precise name. A measurement function exhibits evaluation blindness when the system it measures is failing but the function still produces a value indistinguishable from a non-failing state. Applied to webhook delivery, a log that only records 200 responses, without tracking what those 200s actually preceded, is exhibiting exactly that blindness: it confirms that a response was sent, and says nothing about whether the write it was supposed to guarantee ever happened. Measurement infrastructure, in webhook systems as in AI evaluation pipelines, functions as a correctness concern in its own right rather than an operational afterthought layered on once the core logic is working. A retry policy, a queue, and a dead-letter table add up to reliability only when the observability layer around them can tell whether a system is healthy or only looks that way.