Webhook testing: an integration that survives retries, delays and forgery

A webhook is someone else's service that sends you an event at an unpredictable moment, sometimes twice, sometimes out of order, sometimes forged. Most integrations are checked for "it arrived — we processed it". Defects live in the other cases, and those are what create double orders and stuck statuses.

What a webhook is and why it is unreliable by nature

A webhook is an HTTP request an external service sends to your address on an event: a payment succeeded, a delivery changed status, a subscription renewed. The sender does not know whether you received it, so it repeats on any doubt: a timeout, a network error, your response with an error code. Hence three properties you have to live with: one event may arrive several times, events may arrive out of their order of occurrence, and anyone can send a request to your address if the signature is not verified.

Check checklist

• Idempotency: the same event delivered twice and three times is processed once — by event identifier, not by time of receipt.
• Order: the "payment cancelled" event arrived before "payment created" — the final status is correct, not overwritten by the older event.
• Signature: a request with a wrong or missing signature is rejected and logged; a request with a correct signature but an expired timestamp — as well.
• Response timeout: your handler responds fast and acknowledges receipt, doing the heavy work later; otherwise the sender treats delivery as failed and repeats.
• Processing errors: a temporary database error on your side does not lose the event — it goes to a retry queue, not into the void.
• Non-delivery: the event never arrived — is there a scheduled reconciliation that detects it.
• Format change: the sender added a field or changed the version — the handler does not crash on unknown fields.
• Load: hundreds of events per minute during a promotion — the queue and database hold, processing order does not break.

How to test it in practice

1. Build a request collection in Postman or similar: one example of every event from the real sender, with a signature.
2. Send them by hand and by script to the test stand: one at a time, twice in a row, in reverse order, with a corrupted signature, with an outdated timestamp.
3. Check not the handler's response but the system state afterwards: order status, database records, sent notifications, log entries.
4. Emulate a failure on your side: stop the database or queue at the moment of receipt and make sure the event is not lost.
5. Reconciliation: compare the sender's event list with the list of processed events for the period.

Typical defects

A double order after redelivery of the payment webhook. An order in "paid" after a cancellation because events were processed in order of receipt. A "payment succeeded" push sent twice. A handler that calls an external API inside webhook receipt and misses the sender's timeout. A signature verified only in production because on the test stand it was "temporarily disabled". All of these show up not in a test run but at peak load — when retries and delays are most frequent.

Integrations in the other direction

The same principles apply when you call an external API yourself: timeouts and retries with increasing intervals, idempotency keys when creating payments and orders, handling of rate limits, behaviour on contract changes. The regression suite for outbound integrations is built the same way and run on the same schedule.