Retry with Exponential Backoff and Jitter (and the Retry Budget Nobody Sets)

Retrying with exponential backoff means each failed attempt waits longer than the last — 1s, 2s, 4s, 8s — up to a cap, so a struggling dependency gets progressively more room to recover. Jitter randomizes those delays so that a thousand clients that failed at the same instant don’t all come back at the same instant. You need both: backoff limits how much load a single client generates during an outage, and jitter fixes the shape of that load across all your clients. Neither one, on its own, prevents the failure mode that actually takes systems down.

The short answer

Retry only failures that a retry can plausibly fix. Compute the delay as min(cap, base * 2^attempt), then apply full jitter — pick a random value between zero and that delay. Bound the whole operation with a deadline rather than an attempt count, honour Retry-After when the server sends one, and cap retries globally as a percentage of your normal request volume so that a total outage can’t turn your client fleet into a load generator. The formula is the easy part; the budget and the classification are what keep it from making things worse.

Why naive retries make an outage worse

Here’s the failure I get called in for. A dependency slows down. Every caller times out. Every caller immediately retries. The dependency, which was struggling under normal load, is now receiving double the normal load — then triple, as the retries themselves time out and retry again. What began as a latency blip becomes a hard outage, and it stays down after the original cause has passed, because the retry traffic alone is enough to keep it saturated.

That’s a retry storm, and its defining property is that it is self-sustaining. The system can’t recover during the outage because recovery requires a moment of reduced load, and retries guarantee there is never one. I have watched teams restart a healthy service repeatedly, looking for a bug in it, when the actual bug was in the clients.

Exponential backoff is the first fix. If a client waits 1s, then 2s, then 4s, then 8s, the number of requests it sends during a 60-second outage collapses from dozens to a handful, and the gaps grow wide enough that the dependency gets real idle time to work through its backlog.

But backoff alone leaves a second problem intact. If all your clients failed at the same moment — which is exactly what happens, because they all depend on the same thing — then they all compute the same delay schedule and retry in lockstep. You’ve replaced a continuous flood with a series of synchronized spikes. The dependency comes up, gets hit by 5,000 simultaneous retries, falls over, and the next spike lands 8 seconds later. This is the thundering herd, and it’s why jitter isn’t a refinement. It’s half the mechanism.

Backoff caps the rate; jitter fixes the shape

Jitter means adding randomness to the computed delay so the retries of a client population spread across the interval instead of stacking at its edge. There are three variants worth knowing, and the difference between them matters less than people argue but more than nothing.

Full jitter — sleep = random(0, min(cap, base * 2^attempt)) — spreads retries uniformly across the whole window. It’s the default I reach for. It produces the flattest load curve, at the cost of sometimes retrying much sooner than the nominal schedule.

Equal jitter — sleep = half + random(0, half) where half = min(cap, base * 2^attempt) / 2 — guarantees a minimum wait while still spreading the second half. Use it when retrying too fast has a real cost and you want a floor under the delay.

Decorrelated jitter — sleep = min(cap, random(base, previous_sleep * 3)) — walks the delay up from the previous actual sleep rather than from the attempt number. It tends to recover faster than full jitter after transient blips while still spreading load. AWS’s Architecture Blog popularized this comparison, and it’s worth reading the original rather than the many posts that restate it.

Pick one and be consistent. The thing that actually differentiates a working retry layer from a broken one isn’t which jitter variant you chose — it’s everything in the next three sections.

The implementation

Here’s the shape I ship. It is deliberately boring, and every parameter in it exists because I’ve seen its absence cause an incident.

import random
import time

class RetryableError(Exception):
    """Raised by callers for failures a retry could plausibly fix."""
    def __init__(self, message, retry_after=None):
        super().__init__(message)
        self.retry_after = retry_after  # seconds, if the server told us


def call_with_retry(
    operation,
    *,
    deadline_seconds=30.0,   # total budget for the whole operation
    max_attempts=6,          # backstop, not the primary limit
    base_delay=0.2,
    cap_delay=20.0,
    clock=time.monotonic,
    sleep=time.sleep,
):
    started = clock()
    last_error = None

    for attempt in range(max_attempts):
        try:
            return operation()
        except RetryableError as error:
            last_error = error
        # Anything not RetryableError propagates immediately: it is either a
        # deterministic failure, or one we cannot prove is safe to repeat.

        elapsed = clock() - started
        remaining = deadline_seconds - elapsed
        if remaining <= 0 or attempt == max_attempts - 1:
            break

        if last_error.retry_after is not None:
            # Server-driven backoff wins over anything we computed locally,
            # but still gets jitter so the herd doesn't re-form on its clock.
            delay = last_error.retry_after * random.uniform(0.8, 1.2)
        else:
            ceiling = min(cap_delay, base_delay * (2 ** attempt))
            delay = random.uniform(0, ceiling)   # full jitter

        delay = min(delay, remaining)
        if delay <= 0:
            break
        sleep(delay)

    raise last_error

Three details that are easy to skip and expensive to skip:

clock and sleep are injected so this is testable without real time passing. A retry layer you can’t unit-test is a retry layer whose behaviour under a two-minute outage is a guess.

The deadline is checked and the delay is clamped to what’s left of it. Without the clamp, a client with a 30-second budget can sleep 20 seconds on attempt six and then make a request it has no time to wait for — burning a request on the struggling dependency purely to throw the response away.

Only RetryableError is retried. Everything else propagates. That’s the classification decision, and it deserves its own section.

Classify before you retry

A retry is a bet that the failure was about the world, not about the request. Most bad retry behaviour I see is a client that never made that distinction and retries everything.

Retry: connection failures, 429s, 503s, explicit “try again” errors from the dependency, lock contention, and — carefully — read timeouts. These are conditions where the same request, sent later, has a genuinely different chance of succeeding.

Don’t retry: 400s, 401s, 403s, 404s, validation errors, serialization failures, and anything else that will fail identically on attempt fifty. Retrying a deterministic failure is pure waste: you burn client capacity, you add load to a dependency that is already telling you no, and you delay the error the caller needs to see. This is the same taxonomy that decides what belongs in a dead letter queue — transient failures get the retry budget, deterministic ones get routed out of the retry path immediately.

The dangerous class is timeouts. When a request times out, you don’t know whether the work happened. The server may have processed it completely and failed to get the response back to you. Retrying a timed-out read is fine. Retrying a timed-out write is a correctness decision, not a reliability one — and the only way to make it safe is for the operation to be idempotent, so that a duplicate is a no-op rather than a second charge. That is the precondition for this entire pattern, and I’ve written the long version of it in how to design an idempotent job queue. If your writes aren’t idempotent, adding retries doesn’t make your system more reliable. It makes it wrong more often.

Budget in time, not attempts

max_attempts=5 sounds like a limit. It isn’t a useful one, because it doesn’t tell you how long a caller will wait. With a 20-second cap and five attempts, the worst case is well over a minute — which is fine for a background job and catastrophic for a request with a user attached to it.

Set a deadline for the operation, derive the retries from what fits inside it, and propagate that deadline down the call chain. If the API gateway gives a request 5 seconds and it has already spent 3, the service it calls should be told it has 2 — not allowed to start its own fresh 5-second retry schedule. Attempt counts should exist only as a backstop against pathological loops.

This one change eliminates an entire class of incident where a request that the client abandoned long ago is still being retried by three services downstream, each burning capacity on work whose result nobody will ever read.

The retry budget nobody sets

Here’s the limit that’s missing from almost every retry implementation I review, and the one I’d add first if I could only add one.

Per-request backoff bounds what a single operation does. It says nothing about what your fleet does. During a total dependency outage, every request fails, every request retries the full schedule, and your aggregate outbound volume goes up by a multiple even with perfect backoff and jitter. The client fleet becomes a load generator aimed at a service that is trying to come back.

The fix is a retry budget: cap retries as a fraction of total requests, per client, over a rolling window. Something like 10% — meaning that in steady state, where failures are rare, every retry is affordable, but during a broad outage retries are cut off almost entirely and the failure is surfaced to the caller instead of amplified.

import time

class RetryBudget:
    """Token bucket: retries may not exceed `ratio` of total requests."""

    def __init__(self, ratio=0.1, ttl_seconds=10.0, min_tokens=10.0,
                 clock=time.monotonic):
        self.ratio = ratio
        self.ttl = ttl_seconds
        self.min_tokens = min_tokens   # floor, so low-traffic clients can retry
        self.clock = clock
        self.tokens = min_tokens
        self.updated = clock()

    def _decay(self):
        # Tokens earned in the window expire, so the budget reflects
        # recent traffic rather than all traffic since process start.
        now = self.clock()
        elapsed = now - self.updated
        self.updated = now
        if elapsed >= self.ttl:
            self.tokens = self.min_tokens
        else:
            decayed = (self.tokens - self.min_tokens) * (1 - elapsed / self.ttl)
            self.tokens = self.min_tokens + max(0.0, decayed)

    def on_request(self):
        self._decay()
        self.tokens += self.ratio

    def try_retry(self):
        self._decay()
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False   # budget exhausted: fail fast, don't amplify

Wire on_request() into every attempt and gate every retry on try_retry(). The behavioural change is exactly what you want: an isolated failure retries freely, and a systemic failure stops retrying almost immediately. The min_tokens floor matters — without it, a service that handles two requests a minute can never retry at all.

Retry amplification: retry at one layer

Retries multiply through a call stack. If your gateway retries three times, and the service it calls retries three times, and that service’s database client retries three times, one user request can become twenty-seven requests to the bottom layer. Each layer looks reasonable in isolation. The product is not.

My rule: retry at one layer per failure domain, and make every other layer fail fast. Usually that’s the layer closest to the failure — the one that has the most information about whether the error is transient. Everything above it propagates. When you do need retries at two levels, the outer level’s budget has to account for the inner level’s multiplication, which in practice means the outer level gets one retry, not five.

The same reasoning applies to worker pools pulling from a queue: if the queue already redelivers failed messages with its own backoff, an in-worker retry loop layered on top means each message is being retried at two levels simultaneously. That’s the kind of interaction that only shows up under load, which is why I treat retry topology as an architectural property of the worker fleet rather than a per-client implementation detail.

Let the server drive when it can

A dependency that returns 429 Too Many Requests with a Retry-After header is telling you the answer. Honour it. Server-side backoff signals are strictly better information than anything a client can infer, because the server knows its own recovery state and the client is guessing from a single failed request.

Two caveats. First, still jitter around the server’s value — if a thousand clients all receive Retry-After: 5, obeying it exactly reproduces the thundering herd on the server’s own clock. Multiply it by a random factor near 1. Second, clamp it: a Retry-After of 3600 shouldn’t put your worker to sleep for an hour, it should fail the operation and let the job be rescheduled by the queue.

Backoff also pairs naturally with a circuit breaker, and the two do different jobs. Backoff paces an individual caller’s attempts. A breaker stops the calls altogether once failures cross a threshold, so that a dependency everyone knows is down stops receiving traffic at all. Backoff without a breaker still sends a trickle from every client forever; a breaker without backoff slams the dependency the moment it half-opens. Systems that survive real pressure use both, which is part of the broader argument about designing for disruption.

What to measure

Retry logic is invisible until it’s pathological, so instrument it deliberately:

• Retry ratio — retries as a fraction of total requests, per dependency. This is your early warning; it rises well before error rates do.
• Attempts-to-success distribution — if most successes take three attempts, your first attempt is failing routinely and you’re papering over a real problem.
• Budget exhaustion events — every time the retry budget refuses a retry, something systemic is happening.
• Time spent sleeping in backoff — surfaces the case where callers are waiting far longer than anyone intended.

Alert on the retry ratio crossing a threshold, not on individual retries. A retry is normal; a change in how often you’re retrying is the signal. That distinction — instrument everything, alert on the few things a human should act on — is the whole argument in monitoring vs alerting.

When you don’t need this

If you have exactly one client and a dependency that rarely fails, a fixed short delay and two attempts is genuinely fine — the herd problem needs a herd, and jitter buys you nothing against a population of one.

If the caller is a user waiting on a screen, failing fast and letting them retry deliberately is often better than silently burning their patience in a backoff loop. Put the retries behind the interaction, not inside it.

If your work already flows through a queue that redelivers with its own backoff, don’t add a second retry loop inside the worker; you’ll get multiplication, not resilience. Configure the broker’s schedule instead.

And if your writes aren’t idempotent, retries are the wrong project. Fix that first — otherwise every improvement to your retry logic increases the rate at which you double-apply side effects.

FAQ

What’s a good base delay and cap? Base it on your dependency’s actual recovery behaviour, not a convention. For a fast internal service, a base of 100–200ms with a cap of 10–20s is a reasonable starting point; for a rate-limited third-party API, the cap should be closer to its published window. The cap matters more than the base — it’s what stops exponential growth from producing absurd delays on later attempts.

Full jitter or decorrelated jitter? Full jitter is the safer default and easier to reason about. Decorrelated jitter recovers somewhat faster from brief blips because the delay walks up from the previous actual sleep instead of resetting to the schedule. If you’re choosing between them without a way to measure the difference on your own traffic, take full jitter and spend the effort on the retry budget instead.

Should I retry a timeout? Only if the operation is idempotent, or if it’s a read. A timeout means you don’t know whether the work happened, so retrying a non-idempotent write is a decision to sometimes apply it twice. Make the operation safe to repeat, then retries are free.

How many retries is too many? Any number that lets a systemic outage multiply your traffic. That’s why I bound by deadline and by fleet-wide budget rather than by attempt count — five attempts is fine when 0.1% of requests fail and catastrophic when 100% do.

Do I still need a circuit breaker if I have backoff and jitter? Yes, for prolonged outages. Backoff reduces each client’s rate but never to zero, so a large fleet still delivers steady load to a dead dependency. A breaker takes that to zero and lets it recover, then probes carefully before restoring traffic.

Where should the retry logic live? In one place per failure domain — a shared client wrapper, a service mesh policy, or the queue’s redelivery config — never scattered across call sites. Scattered retry logic is how amplification gets built by accident, one reasonable-looking three-attempt loop at a time.

The rest of the reliability layer these patterns sit inside — idempotency, dead-letter handling, backpressure, and observability that stays honest under load — is collected in my guides on reliable systems, queues and observability, and putting it into a running system is what I do in monitoring and operations work.