Webhook reliability for your own store: retries, idempotency, signature verification, and monitoring
1. Why webhooks are unreliable by nature
A webhook is the other system POSTing an event to your callback URL on its own initiative. It rides on HTTP, and HTTP is connectionless and doesn’t guarantee delivery: a network blip, a restart of your service, or a processing timeout can lose that delivery. Worse, you don’t know whether the other side received it — you sent it but got no 2xx response. Was it lost on the network, or did they process it and the response got lost?
Here’s the counterintuitive judgment:Webhook systems retry by default, so your receiver has to be designed for “this will be delivered more than once,” not built on the assumption that it arrives a single time. Mature platforms like Stripe and Shopify handle it this way: if the receiver doesn’t return a 2xx, they resend with exponential backoff, up to 5 to 8 times over roughly 24 hours.
Without idempotency, the same payment event gets posted to your books several times. For connecting leads into your system, see Website leads to CRM, which faces the same duplicate-event problem.
Miss any one of the four and reliability drops a notch. The table below lays out what each one does and what happens if you skip it.
| Element | Role | Standard practice | What happens if you skip it |
|---|---|---|---|
| Signature verification | Confirm the request really comes from them | Verify the HMAC-SHA256 header | Forged events inject fake orders |
| Fast acknowledgement | Stop them from timing out and resending | Return 2xx first, process asynchronously | Slow processing causes duplicate deliveries |
| Idempotent deduplication | Process duplicate events only once | Write event IDs to a table and dedupe | The same order gets posted multiple times |
| Retry backoff | Temporary failures eventually get through | Exponential backoff with jitter | A transient glitch loses the order outright |
| Dead-letter monitoring | Failed events aren’t dropped silently | Over-limit events go to a dead-letter queue with alerts | Orders go missing for a long time with nobody the wiser |
2. Signature verification: confirm the request really isn’t a forgery
A webhook callback URL is publicly reachable, so anyone can POST data to it. Without signature verification, an attacker can craft a fake “payment succeeded” event and your backend may actually grant access or ship an order. This isn’t a theoretical risk — it’s an attack surface that has been hit in the wild.
The standard approach is HMAC-SHA256: the sender signs the request body with a shared secret and puts the signature in a header (such as X-Signature). When you receive it, you recompute the signature over the raw request body with the same secret and compare the two. Only process the request if they match.
Two details are mandatory. First, compare using a constant-time function, not a plain string ==, to prevent timing attacks. Second, validate the timestamp in the header and reject anything older than a 5-minute tolerance to stop replay of captured requests. And you need to be able to rotate the secret immediately if it leaks.
| Check | How to do it | If it fails |
|---|---|---|
| Does the signature match? | Recompute and compare with HMAC-SHA256 | Reject with 401 |
| Is the timestamp fresh? | Within 5 minutes of server time | Reject to prevent replay |
| Has the secret expired? | Support overlapping old and new secrets during rotation | Retry with the old secret |
| Has the body been tampered with? | Sign the raw bytes | Don’t JSON-decode before signing |
3. Acknowledge fast, process asynchronously: don’t do heavy work in the request
This is the easiest trap to fall into. Many people receive a webhook and then do a chain of heavy work inside the synchronous request: query the database, send email, call third-party APIs, write logs — five seconds all told. The sender waits three seconds, gets no 2xx, marks it as timed out, and starts resending. Meanwhile you’re still processing the first one, and the second one has already arrived.
The right pattern has three stages: on receiving the request, verify the signature (milliseconds), then immediately persist the event as-is and return 2xx, and let a background job process it slowly. That way the sender always gets an acknowledgment within three seconds and won’t resend. The heavy work can take its time — the user already has a receipt. Failed events stay in the queue and the async job retries them on its own.
Example scenario: A payment-succeeded event comes in. The receiver does only three things: verify the signature, write the event ID and raw payload to an events table, and return 200 immediately. The real work — adding inventory, sending email, notifying ERP — is done slowly by a background worker that pulls pending events from the events table. Even if sending email takes 10 seconds, it won’t trigger a resend. For a similar approach to form-delivery reliability, see Inquiry Form Reliability.
| Stage | Action | Time target | On failure |
|---|---|---|---|
| Synchronous receive | Verify signature + persist + return 2xx | Under 300 milliseconds | Reject immediately if verification fails |
| Async processing | Query database, send notifications, call downstream services | Seconds to minutes is fine | Failures stay in the queue for retry |
| Retry | Re-run at backoff intervals | Exponential backoff | Move to dead-letter queue past the limit |
| Manual recovery | Check the dead-letter queue and trigger manually | Daily inspection | Log the reason for manual recovery |
4. Idempotency and retries: the same event ten times is still processed once
Idempotency means running the same operation once or ten times produces exactly the same result. Webhooks must be idempotent by nature, because you can’t stop the sender from resending. The fix is simple: every event the sender fires carries a globally unique event ID. Before you process anything, look up that ID in your events table. If you’ve seen it, just return success and skip the work.
Use exponential backoff with jitter for retries: wait 1 minute after the first failure, 2 minutes after the second, 4 minutes after the third, doubling each time, up to 5 to 8 attempts spanning roughly 24 hours. The random jitter keeps every failed event from retrying in the same second and crushing your service all over again (this is called a retry storm). Events that exceed the retry limit aren’t dropped; they go to a dead-letter queue for someone to handle.
| Retry attempt | Backoff interval (approx.) | Elapsed time | Description |
|---|---|---|---|
| Attempt 1 | 1 minute | 1 minute | Temporary network blip |
| Attempt 2 | 2 minutes | 3 min read | Brief service restart |
| Attempt 3 | 4 min read | 7 min | Downstream temporarily unavailable |
| Attempt 4 | 8 min read | 15 min | Longer outage |
| Attempts 5 to 8 | 15 to 30 minutes | About 24 hours | Still failing, goes to dead-letter queue |
5. Monitoring and dead letters: don’t let lost orders happen silently
The most insidious webhook failure is the silent lost order: the sender thinks it delivered, you have no record at all, and weeks later a customer complains “I paid, why hasn’t it shipped?” before you realize the event never arrived. So your monitoring has to watch three things: delivery success rate, retry queue backlog, and dead-letter queue count.
Set concrete thresholds: if the retry queue backs up past 100 events, or no delivery succeeds for 30 straight minutes, alert your usual channels immediately. Check the dead-letter queue once a day and manually reprocess anything stuck there. If your delivery success rate drops below 99%, investigate. A healthy webhook success rate should be close to 99.9%; falling below 99% means you have a persistent problem. For how to set up general monitoring and alerts, see Site monitoring and alerts.
| Metric | Health score | Alert threshold | Action |
|---|---|---|---|
| Delivery success rate | ≥ 99.9% | Below 99% for 1 hour straight | Check receiver logs |
| Retry queue backlog | Close to 0 | Over 100 events | Check whether downstream is stuck |
| Dead-letter queue | 0 | Any backlog at all | Reprocess manually and find the cause |
| No successful deliveries | Events keep coming | 0 for 30 minutes | Check whether the callback URL is blocked |
| Signature verification failures | Close to 0 | 10x spike | Someone may be scanning your endpoint |
FAQ
Why do I sometimes get duplicate order events on my webhook?
Should I process a webhook immediately after receiving it, or return first?
Can I skip signature verification since it’s all over HTTPS anyway?
How should I set the retry interval, and why not just retry every minute?
What do I do with events that still fail after many retries?
How do I know if webhooks are silently dropping events?
Final thoughts
The core insight about webhook reliability is this: it isn’t a one-way push, it’s a two-way protocol where no acknowledgment means a resend. When you design the receiving end, you must treat duplicate deliveries, forged events, and lost events as the default assumption, not as low-probability edge cases. Of the four elements, idempotent deduplication and fast acknowledgment are the two most easily overlooked and the two that cause the most incidents. Get those two right first, and your incident rate drops by more than half.
Three steps to make it work. First, the receiving endpoint does nothing but verify the signature, write to the database, and return a 2xx within 3 seconds — everything else runs asynchronously. Second, build a dedup table keyed on event ID, and retry with exponential backoff plus jitter. Third, review the dead-letter queue daily, and wire delivery success rate and backlog size into alerts. Do these three things and your callback pipeline goes from “orders occasionally go missing and you hope for the best” to “if one goes missing, you know the same day and can recover it.”
One line worth remembering: webhook reliability isn’t about the sender sending reliably — it’s about you receiving dumbly. Verify the signature, respond fast, dedup aggressively. Better to process something twice than to miss a single order.