Webhook delivery: retries, duplicates and order

Webhooks travel over the open internet, to servers that restart, deploy and fall over. So senders retry, and once retries exist, receivers get the same event more than once, and sometimes in the wrong order. Most webhook providers say plainly that delivery is "at least once" and order isn't guaranteed. This lesson looks at delivery from both sides: a sender that retries, tested against a bin that fails on purpose, and a receiver that stays correct when events repeat or arrive late.

The sender: retrying with backoff

A sender treats a 2xx response as delivered and anything else, or no response at all, as a failure to retry later. To test that without a broken server of your own, point it at a webhook bin and use ?status= to choose the answer. Here the bin fails twice, then succeeds:

// delivery.mjs, part 1: a sender that retries, pointed at a bin that fails twice
const BASE = "https://api.testingapis.com";
const t0 = Date.now();
const log = (...a) => console.log(`+${((Date.now() - t0) / 1000).toFixed(1)}s`, ...a);

const bin = (await (await fetch(`${BASE}/api/webhooks/bins`, {
  method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "delivery-demo" }),
})).json()).data;

// The bin answers with whatever ?status= says; here the first two attempts fail
const statuses = [500, 503, 200];
const event = { id: "evt_1001", type: "order.paid", orderId: 42, version: 3 };

async function deliver(url, event, { attempts = 5 } = {}) {
  for (let attempt = 1; attempt <= attempts; attempt++) {
    const status = statuses[attempt - 1] ?? 200;
    const res = await fetch(`${url}?status=${status}`, {
      method: "POST",
      headers: { "Content-Type": "application/json", "X-Event-Id": event.id, "X-Delivery-Attempt": String(attempt) },
      body: JSON.stringify(event),
    });
    log(`attempt ${attempt}: ${res.status}`);
    if (res.ok) return attempt;
    await new Promise((r) => setTimeout(r, 500 * 2 ** (attempt - 1))); // 0.5 s, 1 s, 2 s ...
  }
  throw new Error("gave up");
}
await deliver(bin.url, event);

+0.3s attempt 1: 500
+0.8s attempt 2: 503
+1.9s attempt 3: 200
+1.9s the bin captured 3 requests: attempt 1 of evt_1001, answered 500; attempt 2 of evt_1001, answered 503; attempt 3 of evt_1001, answered 200

Three attempts, with the waits doubling between them, and the bin recorded all three with the same event ID. That last detail is the point: from the receiver's side, a retry is the same event arriving again. Each attempt also carries its attempt number, so the receiver's logs can show what happened.

What to check on a sender

• Which responses count as delivered: every 2xx, or only 200?
• How long it waits between attempts, how many it makes, and whether the waits grow.
• That a 404 or 410 stops retries or disables the endpoint, rather than retrying forever.
• How long it waits for a slow receiver before counting the attempt as failed.
• Whether every retry carries the same event ID, so receivers can recognise it.

The receiver: fast, idempotent, order-aware

Three habits keep a receiver correct under at-least-once delivery:

// delivery.mjs, part 2: a receiver that processes each event once, and ignores stale ones
import { createServer } from "node:http";

const processed = new Set();
const latestVersion = new Map(); // orderId -> highest version applied

const server = createServer((req, res) => {
  let body = "";
  req.on("data", (c) => (body += c));
  req.on("end", () => {
    const e = JSON.parse(body);
    // Answer first, quickly; do the work afterwards
    res.writeHead(200).end();
    if (processed.has(e.id)) return log(`receiver: ${e.id} already processed, skipped`);
    processed.add(e.id);
    if ((latestVersion.get(e.orderId) ?? 0) >= e.version) return log(`receiver: ${e.id} is version ${e.version}, older than what we have, ignored`);
    latestVersion.set(e.orderId, e.version);
    log(`receiver: applied ${e.id} (${e.type}, version ${e.version})`);
  });
});

Then four deliveries for one order: a payment, a retry of the payment, a shipment, and a creation event that was delayed:

await send({ id: "evt_2001", type: "order.paid", orderId: 7, version: 2 });
await send({ id: "evt_2001", type: "order.paid", orderId: 7, version: 2 }); // a retry of the same event
await send({ id: "evt_2002", type: "order.shipped", orderId: 7, version: 3 });
await send({ id: "evt_2000", type: "order.created", orderId: 7, version: 1 }); // arrives last, belongs first

receiver: applied evt_2001 (order.paid, version 2)
receiver: evt_2001 already processed, skipped
receiver: applied evt_2002 (order.shipped, version 3)
receiver: evt_2000 is version 1, older than what we have, ignored

The retry was recognised by its event ID and skipped. The creation event, version 1, arrived after version 3 and was ignored instead of rolling the order back to "created". Without those two checks the order would have been paid twice and ended up in the wrong state.

Testing a receiver

• Send the same signed event twice. The effect, such as a charge or an email, happens once.
• Send two events for the same resource in reverse order. The final state is the newer one.
• Make the processing slow on purpose, and check the receiver still answers within the sender's timeout.
• Make processing fail after answering 200, and check the event can still be retried from your side, because the sender won't resend it.
• Restart the receiver between two deliveries of the same event, and check the duplicate is still caught.

The fourth case is the uncomfortable cost of answering fast: once you've said 200, the sender considers the event delivered. So the receiver needs its own record of events it accepted but hasn't finished, and a way to retry them. The REST tutorial's lesson on timeouts, retries and rate limits covers idempotency keys, the same idea applied to requests a client sends.