Kafka Retry Topics and Dead Letter Queue Patterns in Production

Not every failure should retry immediately

Consider a consumer processing notification events.

Transient:

email provider HTTP 503

Permanent/deterministic:

template ID does not exist
invalid payload schema

Retrying the second case every second forever wastes capacity.

I classify failures.

Retry topic chain

booking-events
     |
 transient failure
     v
booking-events.retry-1m
     |
     v
booking-events.retry-10m
     |
     v
booking-events.dlq

Each retry carries:

event_id
original_topic
original_partition
original_offset
consumer
attempt
failure_class
first_failed_at

KafkaJS-style routing

try {
  await handler(message);
} catch (error) {
  const attempt =
    Number(
      message.headers[
        'x-retry-attempt'
      ]?.toString() ?? 0,
    );

  if (
    isTransient(error) &&
    attempt < 3
  ) {
    await retryPublisher.publish(
      retryTopic(attempt + 1),
      message,
      {
        ...message.headers,
        'x-retry-attempt':
          Buffer.from(String(attempt + 1)),
      },
    );

    return;
  }

  await dlqPublisher.publish({
    originalTopic: topic,
    eventId: header(message, 'event-id'),
    consumer: 'notification-service',
    attempt,
    errorClass: classify(error),
    errorMessage: safeError(error),
    payload: message.value,
  });
}

DLQ is not the end of the design

A DLQ needs:

• owner;
• alert;
• dashboard;
• reason code;
• replay tooling;
• access control;
• retention;
• runbook.

Without those it becomes a permanent graveyard.

Replay

I prefer selective replay:

filter:
  errorClass = TEMPLATE_NOT_FOUND
  and failedAt >= incident start

dry run:
  count = 1,241

replay:
  target = original topic
  rate limit = 50/sec

A mass replay can recreate the original outage if downstream is not ready.

Ordering

Retry topics can reorder messages.

Example:

event version 10 fails
version 11 succeeds
version 10 returns later

For strict per-aggregate ordering I may instead pause that key/partition, store deferred work, or make handlers version-aware.

Poison message

One malformed message should not block the full partition indefinitely.

After a small deterministic retry/classification, route it away and continue—if business ordering allows.

Production checklist

• Classify transient vs deterministic.
• Bound attempts.
• Add delay between attempts.
• Preserve event identity.
• Include diagnostic DLQ metadata.
• Alert on DLQ rate.
• Assign owner and triage SLA.
• Provide safe selective replay.
• Rate-limit replay.
• Understand ordering consequences.

FAQ

Should DLQ messages retry forever?

No.

Does a DLQ solve the root cause?

No. It isolates work so operations/development can diagnose it.

Are retry topics always required?

No. For short transient failures a small in-process retry may be enough. Retry topics are useful when delay should not block the consumer.

Related architecture guides

• Idempotency

• Transactional Outbox

  Architect's final take

  A DLQ is useful only when somebody owns what happens next. Error isolation without recovery is incomplete architecture.