How to Design Webhooks That Don't Lose Events

Stripe retries a failed webhook delivery for up to three days. GitHub does not retry it even once. If your integration treats those two providers the same way, one of them is quietly losing you events.

Webhooks look like the easiest integration pattern in the business: the provider POSTs JSON at your URL and you act on it. The trap is that HTTP gives you none of the guarantees the mental model assumes. Deliveries fail, arrive twice, arrive out of order, or never arrive at all. Designing for that reality is the whole job, whether you are consuming webhooks or building an API that sends them.

The Contract Is At-Least-Once, Unordered, Duplicated

Every serious provider documents the same three caveats, and most integrations ignore all of them.

First, delivery is at-least-once at best. Retries exist because networks fail, and a retry after a timeout can duplicate an event your server actually processed but was too slow to acknowledge. Stripe’s webhook docs ↗ say it plainly: endpoints “might occasionally receive the same event more than once”.

Second, ordering is not guaranteed. Stripe states it does not deliver events in the order they were generated. A customer.updated can land before the customer.created it logically follows.

Third, delivery is not guaranteed at all. GitHub caps payloads at 25 MB and drops anything larger, and failed deliveries are not retried automatically; you have to notice and redeliver them yourself.

So the contract you are really integrating against is: events arrive zero or more times, in any order. Everything below follows from that sentence.

Receiving Webhooks: Five Rules

Return 2xx before doing anything interesting

Providers enforce short response timeouts, and a slow response counts as a failure. The failure mode compounds: a traffic spike slows your handler, timeouts trigger retries, retries add load, and now the same slow work runs twice in parallel.

The fix is to make the handler do almost nothing: verify, persist, acknowledge.

app.post('/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body,
        req.headers['stripe-signature'],
        process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch {
      return res.status(400).send('Invalid signature');
    }

    await queue.enqueue('stripe-events', {
      id: event.id,
      type: event.type,
      payload: event.data,
    });

    res.sendStatus(200); // acknowledge fast, process later
  }
);

The real work happens in a worker that pulls from the queue, with its own retry policy and its own failure isolation. If you do not already run one, our guide to background jobs and task queues covers the options.

Deduplicate on event ID

Because retries duplicate events, your processing must be idempotent. The mechanism is the same one described in our post on idempotent API endpoints: record every processed event ID and skip repeats.

Do it with a unique constraint, not a lookup:

INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING;

If the insert affected zero rows, you have seen this event; stop. Run the insert in the same transaction as your side effects, so a crash between “marked processed” and “actually processed” is impossible. A check-then-insert in two steps has a race window, and webhook retries are exactly the workload that finds race windows.

Never trust the order

Treat a webhook as a poke, not as a fact. The event tells you something changed about subscription X; the payload’s snapshot of X may already be stale by the time your worker runs, and an earlier event may still be in flight behind it.

The robust pattern is to fetch the current state from the provider’s API when the event arrives and reconcile against that, rather than applying event payloads as deltas in arrival order. It costs one extra API call per event and eliminates an entire class of bugs, including the updated before created case, which simply becomes an upsert.

Verify the signature

A webhook endpoint is an unauthenticated public URL that mutates your database. Without verification, anyone who finds it can forge events.

Providers sign each delivery with a shared secret, typically an HMAC over the raw body, sent in a header. Verify against the raw bytes, before any JSON parsing or body-parsing middleware touches the request, and use a constant time comparison. Signed timestamps close the replay hole: Stripe’s libraries reject signatures older than five minutes by default.

Reconcile with polling

Even a perfect consumer misses events when the outage is on your side and the provider’s retry window expires, or when the provider never retries. A nightly reconciliation job that lists recent objects from the provider’s API and diffs them against your database turns “we silently lost a payment event in March” into “the job healed it overnight”. Webhooks for latency, polling for truth.

Sending Webhooks: The Provider Side

If you are building the API that emits webhooks, you own the other half of the contract.

Persist before you POST. Writing your database record and then firing the HTTP call in the same request handler means a crash between the two silently drops the event. Write the event to an outbox table in the same transaction as the state change, and let a dispatcher deliver from there. This is the transactional outbox pattern, and webhooks are its textbook use case.

Retry with exponential backoff and jitter. A fixed retry interval turns one consumer outage into a synchronised thundering herd. The same schedule logic covered in our post on retry and circuit breaker patterns applies; Stripe’s production schedule stretches retries across three days.

Send thin payloads with stable IDs. Every event needs a unique ID (so consumers can deduplicate), a type, a timestamp, and enough data to fetch the rest. Thin payloads keep you under size caps like GitHub’s 25 MB limit, avoid leaking data to endpoints with stale permissions, and push consumers towards the fetch-and-reconcile pattern that survives reordering anyway.

Sign everything and let consumers rotate secrets. HMAC with a per-endpoint secret, timestamp in the signed material, and support for two active secrets at once so rotation does not require a synchronised cutover.

How much of this providers actually do varies more than you would expect:

Behaviour	Stripe	GitHub
Automatic retries	Up to 3 days, exponential backoff	None; manual redelivery
Ordering guarantee	None	None
Duplicate deliveries	Possible, dedupe on event ID	Possible, dedupe on delivery ID
Payload signing	HMAC, timestamped	HMAC
Payload size cap	Thin events available	25 MB, larger payloads dropped

The GitHub webhook events reference ↗ is worth a skim even if you never integrate with GitHub, purely as a catalogue of how many event types a mature product ends up emitting.

When a Webhook Is the Wrong Tool

Webhooks are for server-to-server notification of discrete events at modest frequency. They are the wrong tool when you need sub-second bidirectional updates to end users, which is WebSocket territory, and overkill when a consumer only needs data once a day, where a scheduled API pull is simpler than operating an always-up public endpoint.

The test is simple: if losing a single event costs you money or correctness, you need everything in this post. If it costs you nothing, you probably do not need webhooks at all.

Build the boring version: verify, enqueue, acknowledge, deduplicate, reconcile. Every incident post-mortem involving webhooks ends with one of those five words.