Webhook integration services

Why webhooks go missing

Recovering after an outage

Senders usually retry failed deliveries for a while; Stripe, for example, retries in live mode for up to three days with increasing gaps, and GitHub advises redelivering missed webhooks once your server is back. Retries do not cover every case, so after any significant outage, list the events the sender recorded during the period and confirm each was processed, redelivering or fetching from the API any that were missed. A written recovery procedure means this happens calmly rather than being improvised.

Record outages and recoveries, so recurring causes can be fixed.

Monitoring webhook health

Reconciliation as a safety net

Even with retries and monitoring, an event can occasionally be missed. A scheduled reconciliation, comparing a summary of your records with the sender's API for a recent period, catches these gaps: payments the sender recorded but you did not, orders with different statuses, records changed on one side only. Run it daily or weekly depending on how quickly differences matter, and send only the mismatches to a person for review.

Reconciliation also reveals bugs in processing logic that monitoring alone would miss.

Rotating secrets without downtime

Webhook signing secrets should be changed periodically and immediately if exposure is suspected. Stripe allows the old secret to remain valid for up to 24 hours during rotation, so your endpoint can be updated without rejecting genuine events. Plan rotations as a routine task, with a checklist, and confirm signature failures stay at zero afterwards.

A simple runbook

1. 01

   Detect

   Alert on failed deliveries or signature errors.

   Output: Problem noticed

2. 02

   Diagnose

   Check endpoint health, certificates, secrets and logs.

   Output: Cause found

3. 03

   Fix

   Restore the endpoint or correct configuration.

   Output: Deliveries resume

4. 04

   Recover

   Redeliver or fetch missed events for the period.

   Output: Records complete

5. 05

   Review

   Record cause and prevention.

   Output: Fewer repeats

Changes that can break webhooks

• Deployments

  New code or hosting changes the endpoint or alters the raw body.

• Framework updates

  Body parsing or security middleware rejects deliveries.

• Sender versions

  New event formats or types appear.

• Domain or certificate changes

  The sender can no longer connect securely.

Checks after every deployment

Many webhook failures start with an unrelated change: a framework update that alters how request bodies are read, a security setting that blocks unexpected requests, or a new hosting route. After each deployment, send a test event and confirm it is verified and processed, and watch delivery failures for the next hour. Including the webhook endpoint in automated deployment tests catches many of these issues before customers notice.

Stripe's documentation, for example, notes that frameworks applying cross-site request forgery checks may need the webhook route exempted.

Who looks after the webhooks

Agree who receives webhook alerts, who can access the sender's dashboard to check and redeliver events, and who owns the secrets. Keep this information in the runbook, and review access when people change roles or providers change.

Several senders

Businesses often receive webhooks from payments, ecommerce, forms and signature services. Keep a list of each sender, its endpoint, secret owner and recovery method, so nothing is forgotten when something changes.

Reporting on webhook health

A short monthly summary of delivery failures, recoveries, reconciliation mismatches and rotations keeps everyone aware of how the integration is performing and highlights recurring causes worth fixing.

Separate test endpoints

Keep test and live webhook endpoints and secrets separate, so test events never touch live records.