Webhooks vs Polling: Picking the Right Trigger

The short answer

The framing as a versus question is what causes the damage. Teams pick one, defend the choice, and inherit exactly the failure mode the other one covers. A webhook only system loses every event that arrived during a deploy longer than the provider's retry window. A polling only system is either expensive or slow, and on most APIs it still cannot see a record that was created and deleted between two polls.

This is integration plumbing rather than anything clever, and it is where most of the reliability in an automation actually comes from. It is the bulk of what systems integration work involves.

What each mechanism actually guarantees

Compare the guarantees rather than the convenience, because the guarantees are what you will be debugging at the end of the quarter.

Two more options exist and get forgotten. Provider delta endpoints, such as a change feed with a cursor, give you polling economics with webhook completeness, at the price of a cursor that expires and forces a full resync. Change data capture, reading the database log directly, gives ordered and complete streams when you own the database, and is the correct answer for internal systems far more often than people expect.

The pattern that makes the choice stop mattering

Run both paths into one handler and treat the webhook as an optimisation over the poll, not as the mechanism. This costs one extra scheduled job and removes the entire category of incident where an integration was silently a day behind.

Receiving a webhook without losing events

The receiver has one job, which is to prove the message is genuine, claim it once, and acknowledge inside the provider's timeout. Every additional thing you do in that request handler is a way to lose events, because providers disable endpoints that fail repeatedly and most of them do it without a conversation.

# receiver.py
# The whole job of this file: prove the message is real, claim it once, ack fast.
# Everything else happens in a worker that this file never waits for.

import hmac, hashlib, time, os
from flask import Flask, request, abort

app = Flask(__name__)
SECRET    = os.environ["WEBHOOK_SECRET"].encode()
TOLERANCE = 300          # seconds. Older than this is a replay, not a late delivery.

@app.post("/hooks/vendor")
def receive():
    raw = request.get_data()                       # BYTES, exactly as sent.
    # Never request.json here. Parsing and re-serialising changes key order and
    # whitespace, the recomputed HMAC stops matching, and you will blame the vendor.

    sig = request.headers.get("X-Signature", "")
    ts  = request.headers.get("X-Timestamp", "0")

    if abs(time.time() - int(ts)) > TOLERANCE:
        abort(400, "stale timestamp")              # replay protection

    expected = hmac.new(SECRET, ts.encode() + b"." + raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):     # constant time. Not ==.
        abort(401)

    event_id = request.headers.get("X-Event-Id") or hashlib.sha256(raw).hexdigest()

    if not claim(event_id):        # INSERT ... ON CONFLICT DO NOTHING, returns rowcount>0
        return "", 200             # already seen. At-least-once delivery is normal.

    enqueue({"event_id": event_id, "body": raw.decode(), "received_at": time.time()})
    return "", 200                 # inside the provider timeout, every time

# The rules this file exists to enforce:
#  1. No business logic, no third-party calls, no model calls, no waiting.
#  2. Return 5xx ONLY when you want the provider to retry. A validation failure
#     you will never accept is a 4xx, or the retry runs until the endpoint is disabled.
#  3. The payload is a notification, not evidence. The worker re-reads the object
#     from the API using the id, because the payload was true when it was built
#     and the object may have changed three times since.

The raw body rule deserves its own paragraph because it burns a day of somebody's life on nearly every integration. Signatures are computed over the exact bytes transmitted. Frameworks that parse JSON for you produce a dictionary, and serialising that dictionary back to a string changes key order, whitespace and unicode escaping. The recomputed hash then differs from the header, every request fails verification, and the natural conclusion is that the vendor's signing is broken. Capture the raw bytes before any middleware touches them.

• Do the work asynchronously. If processing takes longer than the provider's timeout you get a retry for an event you are still handling, and now two workers are writing the same record. Ack, enqueue, return.
• Choose your status codes deliberately. A 5xx means retry me. Return it only when a retry could succeed. A malformed payload you will never accept is a 4xx plus a log line, otherwise the provider retries a permanent failure until it disables your endpoint.
• Handle bursts. A backfill, a bulk import or a provider incident can deliver a day of events in a minute. The queue absorbs that. A handler that writes directly to your database does not.
• Log the delivery, not just the outcome. Store event id, received timestamp, signature result and queue offset before processing. When someone asks whether an event arrived at 14:32, the answer has to come from data rather than inference.
• Rotate secrets without downtime. Accept both the old and new signing secret during the rotation window, then drop the old one. Providers rarely coordinate rotation with your deploy schedule.

Polling without silently missing rows

Polling looks simple and has one classic bug that is nearly invisible in testing. You store the maximum updated_at you have seen and next time ask for everything strictly greater. Any record written in the same second as your high water mark, after your query ran, is never returned again. It is not deleted, not errored, and not missing from any report you look at. It is just absent.

Three more polling traps are worth naming. Soft deletes are invisible to a query filtered on updated_at unless the provider bumps that field on delete, so a record that vanished from the source stays in your store forever. Clock skew means the provider's timestamps and yours disagree, which is why the overlap window exists at all. And hard deletes leave no trace whatsoever, so periodic full key comparison, not just an incremental poll, is the only way to detect them.

The arithmetic that sets your poll interval

Poll frequency is a budget decision with three inputs: how stale the data may be, how much of the rate limit you are willing to spend, and how many requests you are prepared to waste on nothing having changed. Work it out rather than picking five minutes because it sounds sensible.

Two readings of that output matter. A high requests per change figure is the honest argument for adding a webhook, because you are paying for silence. And the rate limit percentage is what you have left for everything else: user triggered actions, retries and the backfill you will run one day. Polling that consumes most of the allowance in steady state leaves nothing for the hour you actually need headroom, which is covered in rate limits, retries and backoff.

One scheduling detail saves real incidents. Do not start every poll on the minute. Ten integrations all firing at 00:00 create a spike that trips your own limits and the provider's, and it recurs at the same instant every hour so it looks like a mysterious periodic failure. Add a fixed per integration offset and a small jitter.

How to choose for a given integration

Pick the primary mechanism from the situation, then add the other one as the backstop. The primary decides your latency and your cost. The backstop decides whether you find out when the primary fails.

Where it goes wrong

Assuming events arrive in order

A create retried twice can land after the update that followed it. If your handler applies whatever arrived last, the record ends up in the older state and stays there. The fix is to compare a version, a sequence number or the object's own updated timestamp, and to ignore any event describing a state older than what you already hold. Re-reading the object instead of trusting the payload solves this for free, which is the strongest argument for thin payloads.

Treating a disabled endpoint as a vendor problem

Providers commonly disable subscriptions after a run of consecutive failures, and the notification goes to whichever address was on the integration account. That is frequently a shared mailbox nobody reads. The integration then works perfectly and receives nothing, indefinitely. Sweep-found work is what detects this, usually days before anyone finds the email.

Retry storms after an outage

When your endpoint recovers, the provider delivers everything it has been holding, and your queue depth goes vertical while your workers hammer a downstream API that has its own rate limit. Cap worker concurrency, back off on 429 responses with the provider's Retry-After header rather than a fixed sleep, and let the queue stay deep for a while. Failing slowly is a design choice, and it is covered further in automation error handling.

Verifying nothing at all

An unauthenticated webhook endpoint is an unauthenticated write API on your production system, and its URL appears in browser network tabs, vendor dashboards and support tickets. If a provider offers no signing at all, put a shared secret in the path, restrict by source address where the provider publishes ranges, and treat everything the payload says as a hint to be verified by reading the real object.

Definitions

Terms used precisely on this page

Webhook

An HTTP request sent by a provider to a URL you registered, telling you that something changed on their side. Delivery is typically at-least-once within a retry window, unordered, and permanently lost once that window expires.

Polling

A scheduled request from your system asking a provider what has changed since a stored cursor. It trades latency and request volume for the ability to reconstruct state at any time without depending on anything having been delivered.

At-least-once delivery

A guarantee that a message will be delivered one or more times, never zero within the retry window, and with no upper bound on duplicates. It obliges the receiver to make repeated processing of the same message harmless.

Reconciliation sweep

A periodic pass that compares your stored state against the provider's current state over a window wider than the provider's retry period, correcting differences. The volume of work it finds measures how much the push path is missing.

Keyset pagination

Paging through results using the last seen value of a stable, unique, immutable sort key rather than a numeric offset, so that rows inserted or updated during the read cannot cause pages to skip or repeat records.

Thin payload

A webhook design in which the message carries only a resource identifier and an event type, and the receiver reads current state from the API. It removes staleness and ordering problems at the cost of one extra request per event.