Rate limits, retries and backoff, explained properly

The short answer: classify, obey the server, jitter, then stop

Retry logic is the most copied and least examined code in any integration. It gets written once during an incident, it looks reasonable, and it stays wrong for years because the conditions that expose it are rare and brief. The failure is rarely the algorithm. It is retrying the wrong class of error, ignoring what the server said, and having no ceiling on the damage a retrying fleet can do.

The four limiter shapes, and why they feel different

A rate limit is not one mechanism. Which one you are behind changes the correct client behaviour completely, and you can usually identify it from how refusals arrive rather than from the documentation.

The terms, used exactly

Rate limit

A server-side rule capping how much work one caller may request in a period. It exists to protect the provider's capacity and its other customers, so treating it as an obstacle to route around is how an account gets suspended.

Exponential backoff

A retry schedule where each wait is multiplied by a constant factor, typically doubling. It reduces pressure on a struggling service, and on its own it still synchronises clients that failed at the same moment.

Jitter

Deliberate randomness added to a retry delay so that clients which failed together do not return together. Full jitter picks uniformly between zero and the current backoff ceiling. It is the single most valuable line in any retry implementation.

Retry budget

A fleet-wide ceiling on retries expressed as a share of total requests. When retries exceed the share, the client stops retrying entirely and sheds load, which stops a struggling dependency being kept down by its own callers.

Thundering herd

The load spike created when many clients retry in unison after a shared failure. It is caused by the recovery logic rather than the original fault, and it is why a service can come back up and immediately fall over again.

Read the response before you decide to retry

The status code tells you whether a retry can possibly help. Getting this table right removes more failed runs than any tuning of delays, because most wasted retries are spent on requests that were never going to succeed.

The Clock Stack, the five timers that govern a retry

Jitter, and the storm you cause without it

Fixed backoff does not spread load, it schedules it. Two hundred workers that all failed at the same instant will all wait exactly two seconds, then all retry in the same tenth of a second. The service that was briefly unwell is now being load tested by its own clients, on a two second cycle, until something gives.

Two randomisation schemes are worth knowing. Full jitter waits a uniform random time between zero and the current exponential ceiling, which is simple and effective. Decorrelated jitter uses the previous delay to set the next upper bound, which spreads a long ladder more evenly and is the one in the code below. Either beats a fixed sequence by a wide margin. Neither has a downside worth discussing.

The client that behaves

This is the whole pattern in one function: classification before anything else, the server's header preferred over local backoff, decorrelated jitter on every wait, a total time budget, and an idempotency key so that a retry after a lost response cannot duplicate a write.

# retry.py
# Four rules: classify first, obey the server, jitter always, budget the whole ladder.

import datetime as dt
import random
import time
from email.utils import parsedate_to_datetime

RETRYABLE = {408, 425, 429, 500, 502, 503, 504}
TERMINAL  = {400, 401, 403, 404, 409, 422}


class Terminal(Exception):  ...   # a retry cannot change this answer
class Exhausted(Exception): ...   # send it to the dead letter store, not round again


def retry_after_seconds(resp, fallback):
    """A server that tells you when to come back has ended the argument."""
    v = resp.headers.get("Retry-After")
    if not v:
        return fallback
    try:
        return float(v)                                   # delta-seconds form
    except ValueError:
        when = parsedate_to_datetime(v)                   # HTTP-date form
        return max(0.0, (when - dt.datetime.now(dt.timezone.utc)).total_seconds())


def call(session, method, url, *, idempotency_key,
         attempts=5, base=0.5, cap=60.0, budget=120.0, **kw):

    kw.setdefault("headers", {})["Idempotency-Key"] = idempotency_key
    sleep_prev = base
    started    = time.monotonic()

    for attempt in range(1, attempts + 1):
        # connect timeout and read timeout are different problems, so set them apart
        resp = session.request(method, url, timeout=(3.05, 27), **kw)

        if resp.status_code in TERMINAL:
            raise Terminal(resp.status_code, resp.text[:500])

        if resp.status_code not in RETRYABLE:
            return resp                                   # success, or yours to handle

        # decorrelated jitter: spreads a fleet out instead of synchronising it
        sleep_prev = min(cap, random.uniform(base, sleep_prev * 3))
        delay      = retry_after_seconds(resp, sleep_prev)

        spent = time.monotonic() - started
        if attempt == attempts or spent + delay > budget:
            raise Exhausted(resp.status_code, attempt, round(spent, 1))

        time.sleep(delay)

Three details in there matter more than they look. The split timeout tuple stops a slow read being treated like a dead host. The elapsed check before sleeping prevents a final wait that overshoots the budget by a minute. And the idempotency key is not optional decoration: without it, every retry on a write is a coin flip, which is the subject of idempotency in automation.

One thing the function deliberately does not do is retry inside a client library that is already retrying. Most HTTP libraries and SDKs ship with retry behaviour enabled by default. Turn it off when you own the loop, or your five attempts quietly become fifteen.

Concurrency, the limit nobody counts

Requests per second is the limit people design for. In-flight requests is the limit that actually bites, because it depends on the provider's latency rather than on your intent. When the endpoint slows down, your parallelism climbs on its own and the limiter refuses you without your request rate changing at all.

• Cap your own parallelism with a semaphore and set the value deliberately. Ten workers hitting an endpoint that normally answers in 200ms behave completely differently when it starts answering in 8 seconds.
• Pace your requests with a client-side token bucket rather than waiting to be refused. Being refused costs a round trip, counts against some limiters, and gives you no information you did not already have.
• Give bulk work its own limiter, separate from interactive work. A backfill should never be able to consume the quota that customer-facing calls depend on.
• Prefer batch endpoints where they exist. One request carrying fifty records almost always costs one unit against the limit rather than fifty.
• Cache anything you fetch more than once per run. The cheapest way to stay under a limit is to stop making requests you already made.

Where a provider publishes its limits, write them into a config file next to the workflow rather than into the retry code, and check them again at each integration review. Limits change with plan tiers and with vendor policy, so a number hardcoded eighteen months ago is a number nobody is checking. This belongs in the same place as the rest of the integration contract during systems integration work.

When you are the one being throttled every day

Persistent throttling is a design problem wearing an incident costume. If you hit the limit most days, no retry policy will fix it, because the work you are asking for genuinely exceeds what you are permitted. Four moves in order of effort.

1. Stop polling and start listeningBiggest saving

   Polling every minute for changes that happen twice a day spends almost all of your quota confirming nothing changed. A webhook removes those requests entirely. The trade is covered in webhooks versus polling.

2. Move to the batch endpointOne afternoon

   Rewrite per-record loops as batch calls where the provider offers them. This is usually the largest single reduction available and it also removes the fan-out that most workflow platforms meter per item.

3. Put a queue in front of the integrationOne or two days

   Give every outbound call to that provider a single queue with a fixed drain rate set slightly below the limit. Bursts become latency instead of errors, and you get one place to change the rate rather than a rate hidden inside every workflow.

4. Ask for more, with evidenceFree, often works

   Providers raise limits for callers who can describe their pattern, show they honour Retry-After, and explain what the traffic does. Arriving with a measured request pattern gets a different answer from arriving with a complaint.