Reliable webhooks require a receiver that verifies the sender, accepts duplicate delivery, tolerates out-of-order events, acknowledges quickly, and then processes, observes, and repairs work. No network offers practical exactly-once delivery across two systems, so the design makes at-least-once delivery safe through durable identity and reconciliation.
This guide is explicitly different from webhooks versus polling. That article helps choose how systems detect change and combine notification with repair. This one assumes a webhook has been selected and specifies receiver implementation: signatures, inbox idempotency, queues, retries, replay, and reconciliation.
Define an event contract
Specify event name, version, identity, creation time, provider account or tenant, subject, and required fields. Separate a stable envelope from type-specific payload. Consumers should not infer type from the presence of an optional property.
Decide whether the event is a notification that prompts a state fetch or a factual snapshot. Notifications reduce payload coupling but add a request; snapshots need stricter compatibility and privacy. Send no data the consumer does not need.
Give events stable identity
Every delivery attempt for the same logical event should carry the same event ID. A separate delivery ID can change per attempt if required. Receivers need a database uniqueness key combining provider, account where necessary, and event.
When a provider lacks a trustworthy ID, derive one carefully from stable business references and type rather than a random timestamp. Document collision risk. Payload hashes alone can change when irrelevant fields or formatting change.
Verify raw signed bytes
Follow the provider’s official mechanism, commonly HMAC over a timestamp and raw body. If raw bytes are signed, verify before JSON re-encoding changes spacing or key order. Use a framework or cryptographic library’s constant-time comparison.
Keep secrets out of URLs, source, and logs. Separate test and production and isolate tenant secrets where the provider model requires it. Record key identifiers, not values. Never invent a private signing algorithm.
Prevent replay
Validate signed timestamp within a documented tolerance and retain event identity so a valid old request cannot repeat business work. A valid signature proves possession of the secret, not freshness by itself.
Keep server clocks synchronized. If authorized historical replay is required, use a separate control or explicit replay mode while idempotency remains active. Do not permanently widen the normal acceptance window.
Rotate keys safely
Support current and previous secrets for a short transition or a key identifier that selects the correct secret. Verify under controlled order, monitor old-key traffic, and remove the old value by deadline. Do not keep an unlimited collection of valid secrets.
Test rotation and rollback. A suspected leak triggers incident response and scoped credential replacement, not a silent configuration edit. Return generic authentication failures while retaining safe internal references.
Separate receipt from processing
The synchronous endpoint should enforce method, content type, size, signature, timestamp, account, and basic schema, then durably insert or detect the event and return the expected acknowledgement. Business rules and outbound calls belong in queued work.
Do not return 2xx before durable acceptance if that status tells the provider to stop retrying. On storage failure, return the documented retryable response. Avoid performing a financial transition in a request that can time out after the side effect but before acknowledgement.
Build a deduplicating inbox
Store provider, account, event ID, type, version, received time, processing state, attempts, and protected payload reference. Enforce a unique constraint and detect duplicates transactionally. A process-local cache is not enough when several servers receive the same retry.
Set a retention policy for raw bodies based on support and privacy needs. After processing, a digest and minimized fields may be sufficient. Restrict inbox access and do not expose full personal payloads in a general support dashboard.
Make business effects idempotent
Inbox deduplication alone is insufficient if a worker changes an order then crashes before marking the event complete. Protect the business transition with a unique application record and local transaction. For external side effects, use an outbox or the destination’s idempotency key where supported.
Give messages, stock actions, and credits identities derived from event and purpose. Retrying a transition into the same valid state should not repeat consequences. The electronic payment integration guide illustrates why this matters around money.
Expect out-of-order delivery
invoice.paid can arrive before invoice.created, or an older update can arrive after a newer one. Use provider sequence or object version where available, fetch current state, or hold a narrowly defined missing dependency. Do not delay every event for a rare possibility.
Enforce allowed state transitions and retain non-applied events with a reason. For snapshots, compare authoritative versions rather than receipt time. Test important event sets under several delivery orders.
Classify failures
Malformed payload or unsupported version may be permanent and belong in quarantine. Database locks and temporary provider outages may deserve retry. Unknown account or failed signature may indicate configuration or attack. One unlimited retry policy cannot handle all classes.
Set attempts, timeout, backoff, and jitter, then move exhausted events to a visible failed or dead-letter state. Alert on oldest age, volume, and material change rather than every individual failure.
Design controlled replay
A replay tool selects events by identity, period, or failure reason, previews count, records a reason, and requests approval for sensitive effects. It runs the same idempotent handler. Do not change event ID or paste payloads manually into a production endpoint.
Audit who replayed what and the outcome. Replaying a completed event should not repeat work; replaying a crash after partial effect should finish safely. Distinguish provider redelivery from internal reprocessing.
Add reconciliation
Events can be missed because of configuration, retention, or prolonged outage even with retries. Compare critical external state through an API or report on a defined schedule. Webhooks provide prompt notification; reconciliation proves completeness.
Create owned differences such as external record absent locally, mismatched state, or aged pending event. Repair them through the same stable identities. This complements the transport decision article without cannibalizing its intent.
Resolve tenant from verified context
Map the signed provider account or credential to a tenant after verification. Never trust an arbitrary tenant_id in the payload. Include provider and account in uniqueness when event IDs are not globally unique.
Carry tenant context explicitly into queued work and clear it between jobs. The Laravel tenant isolation test guide includes signed events for another account as a negative case.
Harden the endpoint
Enforce expected HTTP method, content type, body size, read timeout, and rate controls that account for legitimate retries. IP allowlists can add defense but should not replace signatures where addresses change or identity is not guaranteed.
Exclude sensitive bodies from access logs and debug output. Separate environment URLs. Validate any user-configured outbound callback against SSRF risks and document proxy behavior that may alter signed bytes.
Test hostile sequences
Use documented signature fixtures and cover wrong key, stale timestamp, changed byte, malformed JSON, oversized body, duplicate, concurrency, out-of-order events, worker crash, unknown version, and key rotation. Verify response codes that control provider retry.
Run contract tests when proxies, middleware, SDKs, or JSON handling change. Test job retry, replay, and reconciliation. Avoid experimentation against production unless the provider offers a controlled official tool and the change plan permits it.
Observe the event journey
Measure received, deduplicated, signature-rejected, stored, processed, and failed counts; receipt-to-completion time; oldest event; replay outcomes; and reconciliation differences. Use safe correlation IDs. A dashboard of HTTP 200 responses does not prove business completion.
Integration engineering can own this boundary. Set internal objectives from actual provider capability and your baseline rather than an invented universal delivery promise.
Conclusion: operate a repairable inbox
A reliable receiver authenticates raw bytes, time, and account, stores one identity, acknowledges promptly, processes idempotently, and repairs gaps through replay and reconciliation. It never assumes order or single delivery. Request a webhook receiver review with a signed fixture, event contract, and failure sequence, never a production secret.
