Retries, Dead Letters, and the Plugin Chain

A message-driven system is defined by its failure path. The happy path — part 9 — is table stakes; the question that separates toy integrations from production ones is: when the handler throws, where does the message go? There are only four honest answers: retry it, requeue it, park it somewhere inspectable, or drop it. Convey's Convey.MessageBrokers.RabbitMQ package (from DevMentors' open-source toolkit) implements all four, chooses between them with logic worth reading line by line, and — in one corner — has a trap I want you to know about before it knows about you.

All of it lives in one method: TryHandleAsync inside Internals/RabbitMqBackgroundService.cs.

Retries: Polly, fixed interval, in memory

Every delivery is wrapped in a Polly policy:

// Internals/RabbitMqBackgroundService.cs
var retryPolicy = Policy
    .Handle<Exception>()
    .WaitAndRetryAsync(_retries, _ => TimeSpan.FromSeconds(_retryInterval));

With the sample's retries: 3, retryInterval: 2, a failing handler runs up to four times, two seconds apart. Two properties of this design matter operationally.

It is a fixed interval — not exponential. Interestingly, the toolkit's HTTP client (part 14) does use Math.Pow(2, retry) backoff; the broker side never got the same treatment. Three retries two seconds apart is fine against a deadlocked row; it is useless against a dependency that needs ninety seconds to fail over, and there is no jitter to prevent a thundering herd when a thousand messages fail together.

It is also an in-memory, unacked retry. The message stays outstanding on the channel for the whole retry window. That interacts with prefetch: with qos at 1, a message in its retry loop blocks the queue behind it. Head-of-line blocking is the price of in-place retries; the alternative — republish-with-delay via a TTL queue — is more machinery than this package wanted to be.

There is one exception to the retry rule: if MessageProcessingTimeout fires (the handler raced a Task.Delay and lost), the message is immediately nacked, no retries. A timeout is treated as “this will not get better by waiting,” which is a defensible default — but note the losing handler task is not cancelled; it keeps running, detached, while its message is already nacked.

The escape hatch: exceptions become messages

Before Convey gives up on a message, it offers your code a chance to convert the failure into domain information. You register a mapper:

public class ExceptionToMessageMapper : IExceptionToMessageMapper
{
    public object Map(Exception exception, object message)
        => exception switch
        {
            CustomerNotFoundException e when message is CreateOrder m
                => new CreateOrderRejected(m.OrderId, e.Message, "customer_not_found"),
            _ => null
        };
}

When a handler throws and the mapper returns non-null, the subscriber publishes the rejection event and acks the original message. The failure has been handled — it left the queue and re-entered the system as a first-class fact that other services (and the original caller, in the async-API pattern from part 11) can react to. This is the single best idea in the package: the difference between an exception in a log and a create_order_rejected event on the bus is the difference between an incident and a workflow.

The newer IExceptionToFailedMessageMapper refines this with a FailedMessage wrapper carrying two flags — ShouldRetry and MoveToDeadLetter — so you can declare, per exception type, whether the failure is transient (retry first), terminal-but-informative (publish and ack), or terminal-and-suspicious (publish and dead-letter). The decision tree in TryHandleAsync is dense, but it honors those flags faithfully; the legacy mapper is consulted only as a fallback.

The dlx- safety net

When retries are exhausted and no mapper rescues the message, the dead-letter machinery takes over — if you enabled it:

"deadLetter": { "enabled": true, "prefix": "dlx-", "declare": true }

At subscription time, alongside your queue, the initializer declares a direct exchange named {prefix}{exchange} (so dlx-orders), a companion dlx- queue, and sets the x-dead-letter-exchange arguments on your primary queue. A message that gets BasicNacked with requeue: false is then routed by the broker itself into the dead-letter queue, where it waits — with a TTL that the code coerces to 24 hours if you configure it as zero or negative — for a human or a replay tool. Notably, the DLX queue's own dead-letter target points back at the primary exchange, which makes “replay everything in the parking lot” a matter of expiring it.

Now the part that gets teams in trouble. The final nack looks like this:

channel.BasicNack(args.DeliveryTag, false, _requeueFailedMessages);

requeueFailedMessages defaults to false. Walk the two configurations:

• Dead-letter disabled, requeue false (the default if you configure nothing): the terminal nack discards the message. You get an error log line and nothing else. The queue drains; the data is gone.
• Requeue true: the message goes back to the front of the queue. With prefetch 1 and a deterministic failure, you have built a hot loop — the same poison message, forever, while everything behind it starves.

The only correct production posture is dead-letter on and requeue off. The package supports it fully; it just doesn't force it, and the failure mode of forgetting is silence.

The plugin chain

Every delivery flows through a middleware pipeline before reaching your handler. IRabbitMqPlugin is the RabbitMQ-specific middleware contract; plugins are registered at setup:

.AddRabbitMq(plugins: p => p.AddJaegerRabbitMqPlugin())

and composed Russian-doll style — the registry wires each plugin's successor so that calling _successor(...) descends toward the real handler, exactly like ASP.NET Core middleware but for message processing. It is the obvious seam for message-level concerns: tracing, payload logging, schema validation, metrics, idempotency guards.

Which brings me to the trap. The stock Jaeger plugin (part 21 covers it properly) wraps the handler in a span, and when the handler throws it tags the span as errored — and does not rethrow. Sit with that. With the plugin in the chain, TryHandleAsync never sees the exception. No retries. No exception-to-message mapping. No dead-letter. The message is acked as if it succeeded, because from where the ack logic sits, it did. An observability feature silently converts every handler failure into a success with a red span nobody is looking at.

The general lesson outranks the specific bug: anything you put in a message pipeline inherits responsibility for the failure semantics of everything below it. A plugin that catches must rethrow; a plugin that logs must not swallow. When I write pipeline middleware now, the catch { throw; } discipline is the first thing I test, precisely because of the afternoon I spent discovering an empty dead-letter queue that should have had four thousand messages in it.

The scorecard

What this failure path gets right: manual acks everywhere, a real decision tree instead of a blanket policy, the exception-to-message idea (steal it), per-failure-type retry/dead-letter flags, and broker-native dead-lettering rather than a homemade parking table.

What to watch: fixed-interval retries with no jitter, head-of-line blocking during retry windows, a default configuration that discards terminal failures, an uncancelled timeout race, and a tracing plugin that can defeat the entire apparatus.

At-least-once delivery plus manual acks means one more thing is non-negotiable: your handlers will occasionally run twice, so they must be idempotent. The chassis's answer to that — the inbox, and its partner the outbox — is where the next two parts go.