System Design Day 13: Message Queues

Message Queues: Put Slow Work Behind a Buffer

Learning outcome: By the end of Day 13, you will be able to explain how a message queue separates producers from consumers, calculate backlog growth, and identify the reliability work that a queue does not remove.

This lesson is part of the System Design in 30 Days roadmap.

The problem: one slow dependency blocks the request

Imagine that checkout must create an order, charge the customer, and send a confirmation email. If checkout calls the email provider directly before responding, a slow or unavailable email provider can increase customer latency or cause the request to fail even after the important order work succeeded.

Email is normally follow-up work. The customer needs a correct order result immediately; the email can be delivered moments later. A message queue lets the system move that work out of the synchronous request path.

The mental model: a waiting line between two speeds

A message queue is a waiting line between a producer and one or more consumers:

1. Checkout, the producer, creates an email job.

2. The producer publishes the job to the queue.

3. The queue holds the job until a consumer can receive it.

4. An email worker, the consumer, sends the email.

5. The worker acknowledges successful processing according to the broker's protocol.

The producer and consumer no longer need to run at the same moment or at the same speed. That is the core benefit: temporal decoupling plus a visible buffer.

This does not mean every queue behaves identically. Ordering, durability, redelivery, retention, and delivery guarantees depend on the technology and configuration. For example, standard Amazon SQS queues use at-least-once delivery, so consumers must tolerate the possibility of receiving a message more than once.

Worked example: calculate the backlog

Assume:

The backlog growth rate is:

arrival rate - processing rate = 100 - 80 = 20 jobs/minute

After five minutes:

20 jobs/minute x 5 minutes = 100 waiting jobs

We can verify the same result independently:

• Jobs created: 100 x 5 = 500

• Jobs processed: 80 x 5 = 400

• Jobs still waiting: 500 - 400 = 100

The totals reconcile: 400 processed jobs plus 100 waiting jobs equals all 500 jobs created.

The queue has absorbed the spike, so checkout did not have to wait for each email. But the backlog is growing. If the arrival rate stays above processing capacity, the queue eventually reaches a retention, storage, age, or business-latency limit.

What to monitor

Queue depth alone is useful but incomplete. A healthy design usually monitors:

• Backlog size: how many messages are waiting.

• Oldest-message age: how long the oldest job has waited.

• Arrival and completion rates: whether producers are outrunning consumers.

• Processing failures and retries: whether consumers are repeatedly failing the same work.

• Dead-letter volume: how many messages exceeded the normal retry policy.

• Consumer saturation: whether adding consumers would actually increase throughput.

Oldest-message age is especially important because a queue can be small while a blocked or poisoned message waits too long.

Reliability responsibilities do not disappear

Adding a queue changes where failures happen; it does not eliminate them.

Make consumers idempotent

With at-least-once delivery, a worker may receive the same job again. The consumer should be safe to run more than once. For an email job, you might store a stable operation ID and record that the confirmation was already sent before repeating the external side effect.

Acknowledge only after success

If the worker acknowledges too early and then crashes, the job can be lost from the processing flow. If it never acknowledges, the broker may redeliver the job. The exact mechanism varies, so match acknowledgement behavior to the queue you use.

Use bounded retries

Retries help with temporary failures, but unlimited immediate retries can create a hot loop. Apply a retry policy with delay or backoff, then route persistently failing work to a dead-letter queue for inspection.

Connect database state and message publication safely

A common failure occurs when checkout commits the order but crashes before publishing the email job, or publishes the job and then rolls back the order. Patterns such as a transactional outbox can close that gap by recording the event with the business transaction and publishing it reliably afterward.

Try this today

Keep the arrival rate at 100 jobs per minute. Now run two workers that each process 60 jobs per minute.

Total processing capacity is:

2 workers x 60 jobs/minute = 120 jobs/minute

The net change is:

100 arriving - 120 completed = -20 jobs/minute

An existing backlog therefore drains by 20 jobs per minute until it reaches zero. Once the queue is empty, consumers cannot process jobs that have not arrived, so the backlog remains at zero rather than becoming negative.

A completed design answer you can reuse

I would keep order creation synchronous, then publish an email job to a durable queue. Email workers consume independently, acknowledge only after successful processing, and use an idempotency key so redelivery does not send duplicate confirmations. I would monitor arrival rate, completion rate, queue depth, oldest-message age, retries, and dead-letter volume. If backlog age rises, I would scale consumers when the downstream email provider can accept more traffic; otherwise I would apply backpressure or rate controls and protect the dependency.

Common mistakes

• Saying a queue makes work faster. It usually moves work out of the request path; total processing still takes resources and time.

• Treating the queue as infinite. Retention, storage, latency, and operational limits still exist.

• Assuming exactly-once processing. Delivery guarantees vary, and external side effects still need idempotency.

• Ignoring poison messages. A repeatedly failing job needs bounded retries and a dead-letter path.

• Scaling consumers without checking the downstream dependency. More workers can overwhelm the email provider or database.

• Claiming all queues are strict FIFO. Ordering depends on the product, queue type, partitioning, and configuration.

Knowledge check

1. If 150 jobs arrive per minute and consumers complete 120, how quickly does the backlog grow? 30 jobs per minute.

2. Why should a consumer be idempotent? Because retries or at-least-once delivery can cause the same message to be processed again.

3. Which metric reveals how long work has waited? The age of the oldest message.

4. Does a queue fix sustained overload by itself? No. A sustained capacity deficit keeps growing the backlog.

Continue learning with Korshub

Message queues are one building block in event-driven systems. Continue with The Complete Microservices & Event-Driven Architecture course to connect queues with events, retries, service boundaries, and failure handling.

Navigation

• Previous: Day 12 - Consistent Hashing (publish when live)

• Roadmap: System Design in 30 Days

• Next: Day 14 - Publish-Subscribe (publish when live)