A dead letter queue is where messages go when your consumer has given up on them, and the patterns that make it useful are all about what happens after that: classifying failures so only the hopeless ones land there, wrapping each message in an envelope with enough context to debug it, isolating DLQs so one bad producer can’t hide another’s failures, and building a redrive path that replays messages deliberately instead of blindly. The pattern that matters most is the one nobody writes down: a DLQ is an operational loop you commit to staffing, not a checkbox you enable. A dead letter queue that nobody drains is just a slower, more expensive way to lose data.
The short answer
Route a message to the DLQ when retrying it can no longer plausibly help — either because you’ve exhausted a bounded retry budget on a transient failure, or because you’ve classified it as deterministic and there’s no point retrying at all. Store it with its full failure context, not the bare payload. Give each consumer its own DLQ so failures stay attributable. Alert on the rate of dead-lettering rather than the depth. And build redrive as a real tool — rate-limited, sampled, and only run after the root cause is fixed.
Classify the failure before you route it
Every DLQ design decision follows from one question: why did this message fail? Teams that skip the classification step end up with a DLQ that mixes a downstream outage’s worth of perfectly good messages together with genuinely corrupt ones, and then they can’t safely replay any of it.
Three classes cover nearly everything I see in production:
Transient failures are the ones that will probably succeed on a retry — a timeout, a 503 from a dependency, a lock contention error, a rate limit. The message is fine; the world was briefly hostile. These deserve the full retry budget with exponential backoff and jitter, and they should only reach the DLQ after that budget is exhausted.
Poison messages are deterministic. A malformed payload, a schema the consumer can’t parse, a foreign key that references a row that will never exist, a field with a value the code has no branch for. These fail identically on attempt one and attempt fifty. Retrying them is pure waste — you burn worker capacity and delay every healthy message behind them, and you buy nothing. Poison messages should go straight to the DLQ on first detection.
Systemic failures look like poison in the small and transient in the large: every message is failing because you just deployed a bug or a downstream service is entirely down. The dangerous property here is that a naive retry-then-dead-letter pipeline will happily shovel your entire working set into the DLQ during a ten-minute outage. That’s the failure mode that turns a brief incident into a day of recovery work.
The routing rule falls straight out of this taxonomy: retry the transient, dead-letter the poison immediately, and stop the line on the systemic. In practice that means the consumer needs to distinguish an exception it can’t parse (poison) from an exception the network threw (transient), and a circuit breaker or a global failure-rate check needs to pause consumption when nearly everything is failing rather than draining the queue into the DLQ.
class Poison(Exception):
"""Deterministic — will never succeed. Do not retry."""
def consume(msg):
try:
payload = schema.parse(msg.body) # bad shape -> Poison
except SchemaError as e:
raise Poison(f"unparseable payload: {e}")
try:
apply(payload)
except (Timeout, ServiceUnavailable, RateLimited):
raise # transient -> retry with backoff
except ValidationError as e:
raise Poison(f"invalid business state: {e}")
def worker_loop(msg):
try:
consume(msg)
msg.ack()
except Poison as e:
dead_letter(msg, e, attempts=msg.attempts, reason="poison")
msg.ack() # don't leave it to redeliver
except Exception as e:
if msg.attempts >= MAX_ATTEMPTS:
dead_letter(msg, e, attempts=msg.attempts, reason="retries_exhausted")
msg.ack()
else:
msg.retry(delay=backoff_with_jitter(msg.attempts))The important detail is that reason field. It’s the difference between a DLQ you can triage and a pile you have to hand-inspect: “retries_exhausted” during an outage window is a strong candidate for bulk replay, while “poison” almost never is.
The dead-letter envelope: never store the bare payload
The single most common DLQ mistake I find is dead-lettering the original message and nothing else. Six hours later somebody opens the queue, finds a JSON blob, and has no idea which consumer rejected it, why, how many times it tried, or whether it’s safe to replay. The message is technically preserved and practically useless.
Wrap it in an envelope:
def dead_letter(msg, error, attempts, reason):
dlq.put({
"schema_version": 1,
"reason": reason, # poison | retries_exhausted | systemic
"payload": msg.body, # the ORIGINAL, unmodified, for replay
"message_id": msg.id,
"idempotency_key": msg.key, # so replay is safe
"source_queue": msg.queue,
"consumer": CONSUMER_NAME,
"consumer_version": BUILD_SHA, # which code rejected it
"attempts": attempts,
"first_seen_at": msg.first_seen_at,
"failed_at": now(),
"error_type": type(error).__name__,
"error": str(error),
"stacktrace": traceback.format_exc(),
"trace_id": current_trace_id(), # jump straight to the logs
})Two fields do the heavy lifting and are the ones teams leave out. consumer_version tells you whether a fix has already shipped since the message failed — the difference between “replay this now” and “this will fail again the instant you touch it.” And trace_id connects the dead letter back to the actual request trace, which turns debugging from archaeology into a single click. Keep the original payload byte-for-byte separate from the metadata, too; if you flatten your diagnostic fields into the payload, replay quietly sends the wrong message shape back into the main queue.
Isolate DLQs by consumer, not by company
The shared, one-big-DLQ-for-everything design is tempting because it’s one queue to monitor. It fails for a simple reason: it destroys attributability. When the DLQ alarm fires, nobody knows whose it is, so it becomes nobody’s. Meanwhile a steady trickle of expected failures from one low-stakes consumer raises the baseline enough that a genuine spike from a critical consumer disappears into the noise.
Give each consumer (or each queue-consumer pair) its own DLQ. The blast radius of a bad deploy stays contained, each DLQ has an owning team, the alert threshold can be tuned to that consumer’s normal failure rate, and replay targets exactly the messages that belong to the fix you just shipped. In multi-tenant systems, consider going a level further and isolating by tenant as well — one customer sending malformed data shouldn’t be able to fill a DLQ that another customer’s failures need to be visible in. That’s the same isolation logic I apply to worker fleets and failure taxonomies: failures should be contained to the smallest unit that can be reasoned about independently.
Redrive is a tool you build, not a button you press
Redrive — replaying messages from the DLQ back into the main queue — is where the DLQ pattern either pays off or blows up. The blow-up looks like this: the DLQ has 40,000 messages, somebody ships a fix, somebody clicks “redrive all” in the console, and 40,000 messages hit a service sized for normal traffic. Either the fix wasn’t actually the fix and everything re-poisons the DLQ, or the replay itself becomes the second outage.
Replay is a controlled operation with four properties:
- It happens after the root cause is fixed, never before. Blind replay re-poisons by definition — the messages failed for a reason, and if the reason still exists, all you’ve done is take a lap.
- It’s sampled first. Replay 1% or a hundred messages, watch them succeed, then continue. This is the cheapest possible test of “was that actually the fix.”
- It’s rate-limited. Feed messages back at a rate the consumer and its downstream dependencies can absorb on top of live traffic, not instead of it.
- It’s filtered. Replay by
reason,consumer_version, and time window — the messages that failed during Tuesday’s outage, not everything that has ever failed.
def redrive(dlq, *, since, until, reason, limit, rate_per_sec, dry_run=True):
replayed = 0
for env in dlq.scan(since=since, until=until):
if env["reason"] != reason or replayed >= limit:
continue
if dry_run:
log.info("would replay", id=env["message_id"], err=env["error_type"])
else:
# idempotency_key means a double-replay is harmless
main_queue.put(env["payload"], key=env["idempotency_key"])
dlq.delete(env)
replayed += 1
sleep(1 / rate_per_sec)
return replayedNote the dry_run default and the idempotency key. Replay is only safe at all because the consumers are idempotent — a message that partially succeeded before failing will be re-processed, and if that double-charges someone, your DLQ is a liability rather than a safety net. Idempotency is the precondition for this entire pattern; I’ve written the full version of that argument in how to design an idempotent job queue.
Alert on the rate, page on the ownership
DLQ depth is the metric everyone graphs and the wrong one to alert on. A DLQ holding 200 messages from a fixed incident last week is fine. A DLQ that just went from 0 to 40 messages in a minute is an incident in progress, and it might not cross a depth threshold for another hour.
The three signals worth wiring up:
- Dead-letter rate (messages/minute, and as a fraction of throughput) — the leading indicator of a bad deploy or a downstream failure. This is the one that pages.
- Age of the oldest message — the indicator that the DLQ isn’t being drained, which is an organizational failure rather than a technical one. This is the one that files a ticket.
- A non-zero DLQ that nobody has looked at in N days — because that’s the state the whole pattern is meant to prevent.
Set the DLQ’s retention period longer than the source queue’s. This one bites people: if the main queue keeps messages for four days and the DLQ also keeps them for four days, a message that spent three days retrying gets one day in the DLQ before it silently expires — and expiring out of a dead letter queue is exactly the data loss the DLQ existed to prevent. And keep the alerting deliberately quiet: the point is a signal a human will actually act on, which is the same discipline as keeping monitoring separate from alerting.
Where dead letter queues don’t come for free
Kafka has no native DLQ. It has no per-message acknowledgement or redelivery counter, so “dead letter” means a topic you produce to yourself when your consumer’s error handler gives up, and the retry count has to be tracked in a header you maintain. Partition ordering also means a poison message blocks its partition entirely until you either skip it or route it out — which makes fail-fast poison detection more urgent on Kafka than on SQS, not less.
Ordering-sensitive consumers. If your consumer relies on strict per-key ordering, pulling one message out to the DLQ and letting the next one through has just reordered your stream. For ordered work you usually need to halt that key’s processing rather than skip ahead — which means your DLQ needs to carry the whole key’s backlog, or you need to accept the reordering explicitly. This is a design decision, not an implementation detail, and it’s the one most likely to be discovered in production. Graceful degradation under this kind of pressure is the broader subject of designing for disruption.
When you don’t need this
If your queue processes low-volume, low-stakes work and a lost message costs nothing, the full apparatus is overkill — a bounded retry and a log line may genuinely be enough. If you’re on a managed queue with a built-in DLQ and your failure volume is low enough to inspect by hand, you don’t need custom redrive tooling; you need to actually look at the queue. And if your messages are perfectly reconstructible from an upstream system of record, replay-from-source can be simpler and more honest than maintaining a DLQ at all.
The machinery here earns its keep when messages represent work you cannot recreate, failure is constant enough that manual handling doesn’t scale, and the cost of silently dropping a message is real. That’s most systems doing meaningful background work — but it’s worth checking that it’s yours before building it.
FAQ
What’s the difference between a retry and a dead letter queue? Retries are the trying-again mechanism; the DLQ is the giving-up mechanism. Retries handle transient failures that will plausibly succeed on another attempt. The DLQ catches what’s left — messages that exhausted the retry budget, and poison messages that should never have been retried at all — so they can be inspected and replayed instead of being dropped or cycled forever.
How many retries before dead-lettering? For transient failures, commonly 5–8 attempts with exponential backoff and jitter, tuned to how long your downstream typically takes to recover. For messages you’ve classified as poison, zero — retrying a deterministic failure is pure waste. The number matters less than the classification, which is why the taxonomy comes first.
Should I automatically redrive from the DLQ? Not blindly. Automatic replay of messages whose root cause is still present just re-poisons the queue in a loop. Automated redrive is defensible only for narrowly-scoped, provably transient reasons (say, a dependency-outage window that has been confirmed resolved) and only with rate limiting. The default should be a human deciding, with a dry run first.
Does Kafka have a dead letter queue? Not natively. You implement it as a separate topic your consumer’s error handler produces to, tracking attempt counts in message headers yourself. Because a poison message blocks its partition, detecting and routing poison quickly matters more on Kafka than on brokers with per-message redelivery.
What should the DLQ retention period be? Longer than the source queue’s — otherwise a message that spent most of its life retrying can expire out of the DLQ before anyone triages it, which is the exact data loss the DLQ was supposed to prevent.
The full set of reliability patterns these fit into — idempotency, retries, backpressure, and observability that stays honest under load — is collected in my guides on reliable systems, queues and observability, and the day-to-day version of it is what I do in monitoring and operations work.