How to Design an Idempotent Job Queue (Retries, Backoff, and Dead Letters)

If you’re designing a job queue, the single decision that determines whether it survives production is idempotency: processing the same job twice has to produce the same result as processing it once. That’s not a nice-to-have. In any real queue, jobs will be delivered more than once, so the only safe design is one where duplicates don’t corrupt anything. This guide covers how I build that — idempotency, retries with backoff and jitter, and dead-letter handling — with the code that makes it concrete.

The short answer

Assume every job can run more than once, and make that safe. Give each job a stable identifier, make its side effects conditional on whether that identifier has already been applied, retry transient failures with exponential backoff and jitter, and route the jobs that keep failing to a dead-letter queue instead of retrying them forever. Do those four things and the queue absorbs constant small failures without either losing work or spiraling.

Why idempotency isn’t optional

The reason comes down to delivery guarantees. Almost every practical queue offers at-least-once delivery, not exactly-once. Exactly-once across a network and a crash boundary is famously close to impossible in the general case, so real systems guarantee that a message is delivered at least once and accept that it might be delivered more than once.

Concretely, duplicates happen when: a worker finishes the work but crashes before acknowledging it, so the broker redelivers; a visibility timeout expires because the job ran slowly, so the broker hands it to a second worker; or a network blip drops the ack. None of these are edge cases — they’re the normal operating conditions of a distributed queue.

So the question is never “can I prevent duplicates?” It’s “when a duplicate happens, does anything break?” If running a job twice double-charges a customer, sends two emails, or double-counts a metric, you don’t have a reliability problem you can retry your way out of — you have a correctness bug that fires under load. Idempotency is how you make retries safe, and without safe retries you can’t survive at scale.

How to make a job idempotent

The core technique: give every job a stable, unique key, and make the side effect conditional on whether you’ve already applied that key. Two patterns cover most cases.

Dedup on a processed-key table. Record the key of every job you’ve completed, in the same store the side effect touches, inside the same transaction:

def handle(job):
    with db.transaction():
        # If we've seen this job's id, we already did the work. Stop.
        inserted = db.execute(
            "INSERT INTO processed_jobs (job_id) VALUES (%s) "
            "ON CONFLICT (job_id) DO NOTHING",
            [job.id],
        ).rowcount
        if inserted == 0:
            return  # duplicate — already handled

        apply_side_effect(job)   # runs at most once per job_id

Because the dedup record and the side effect commit together, a crash either rolls both back (safe to retry) or commits both (the duplicate is caught). The one rule that matters: the dedup marker must live in the same system of record as the effect, or a crash between them reopens the exact hole you’re trying to close.

Make the effect naturally idempotent. Even better when you can: design the operation so repeating it is a no-op. Prefer upserts over inserts, set-to-a-value over increment, “ensure state X” over “do action that changes state.” An operation that’s idempotent by construction doesn’t need a dedup table at all:

-- running this twice leaves the same row; an INSERT would duplicate
INSERT INTO order_status (order_id, status)
VALUES ($1, 'shipped')
ON CONFLICT (order_id) DO UPDATE SET status = excluded.status;

For genuinely non-idempotent external effects — charging a card, calling a third-party API — pass an idempotency key to the downstream service (most payment and messaging APIs support one) so the provider dedupes for you.

Retries with backoff and jitter

Once jobs are safe to repeat, retrying transient failures is free upside. But retry deliberately, not in a tight loop.

Use exponential backoff: wait longer after each successive failure, so you stop hammering a service that’s already struggling. And add jitter — randomness on the delay — so a thousand jobs that failed at the same instant don’t all retry at the same instant and re-create the outage (the “thundering herd”):

import random

def retry_delay(attempt, base=1.0, cap=300.0):
    # exponential growth, capped, with full jitter
    backoff = min(cap, base * (2 ** attempt))
    return random.uniform(0, backoff)   # e.g. attempt 0->~1s, 4->up to ~16s

Bound the attempts. A job that has failed eight times is not going to succeed on the ninth by magic — something is actually broken, and continuing to retry just hides it while burning capacity. A common shape is 5–8 attempts with growing backoff (say 5s, 15s, 45s, 2m, 5m…), then give up. Which brings us to where “give up” goes.

Dead-letter queues: the giving-up mechanism

A retry is the trying-again mechanism; a dead-letter queue (DLQ) is the giving-up mechanism. When a job exhausts its retries, it moves to the DLQ instead of being dropped or retried forever. The DLQ is where poison messages — jobs that will never succeed, because of a bug or malformed payload — go to be looked at by a human instead of endlessly cycling.

The thing that makes a DLQ useful rather than a graveyard is context. Store enough to actually recover:

def to_dead_letter(job, error, attempts):
    dlq.put({
        "payload":     job.payload,
        "job_id":      job.id,
        "attempts":    attempts,
        "last_error":  repr(error),
        "stacktrace":  traceback.format_exc(),
        "failed_at":   now(),
        "source_queue": job.queue,
    })

With the payload, error, stack trace, and retry count preserved, debugging a failed job is a repeatable process instead of an archaeology dig. And once you’ve fixed the bug, you can replay the DLQ — which only works because the consumers are idempotent, so replaying a job that partially succeeded the first time is safe.

Monitor the queue, not just the workers

A queue has failure signals that server uptime can’t see, and the most important one is DLQ growth. A dead-letter queue that’s filling up is the leading indicator that something systemic broke — a bad deploy, a downstream dependency down, a malformed batch. Alert on the rate of dead-lettering, not just its size.

Watch retry rates too, and correlate spikes with downstream errors: a burst of retries that lines up with a dependency’s error rate is a transient blip riding out backoff, while a burst with no matching cause is more likely a bug you just shipped. And watch queue depth and age — a backlog that’s growing means consumers can’t keep up, which is a capacity or backpressure problem, not a retry problem. (I’ve written more on keeping monitoring separate from alerting so these signals don’t drown in noise.)

When you don’t need all this

Not every queue needs the full apparatus. If your jobs are already naturally idempotent (pure upserts), a modern managed queue may give you retries and a DLQ out of the box, and you mostly need to not fight it. If you’re processing a handful of low-stakes jobs where a rare duplicate is harmless, dedup tables are overkill. The machinery here earns its place when duplicates would cause real damage and volume is high enough that failures are constant — which, for anything doing meaningful background work at scale, they are.

FAQ

What does it mean for a job to be idempotent? Running it more than once produces the same result as running it once. You achieve it either by making the side effect a no-op on repeat (upserts, set-to-value) or by recording a unique job key and skipping the work if you’ve already seen it — with the record and the effect committed together.

Why not just use exactly-once delivery? Because true exactly-once across network and crash boundaries is effectively impossible in the general case. Real queues give you at-least-once and let you build effective-once behavior on top by making consumers idempotent. Chasing exactly-once at the transport layer is the wrong place to solve it.

How many times should a job retry before dead-lettering? Enough to ride out transient faults, not so many that you hide real bugs — commonly 5–8 attempts with exponential backoff and jitter, tuned to your downstream SLAs and the cost of delay. After that, dead-letter it with full context for a human.

What should go in a dead-letter queue message? The original payload, the job id, the attempt count, the last error and stack trace, a timestamp, and the source queue. That’s enough to debug the failure and safely replay it once fixed — replay being safe precisely because the consumers are idempotent.

Designing or fixing the queue and worker layer under a system that keeps losing or double-processing work is a big part of what I do. If your pipeline breaks under load or hides its own failures, here’s how I work with people — or read how I think about worker fleets that survive scale.