7 — Queues and asynchronous work

Not everything has to happen while a user waits. When someone places an order, the payment probably has to complete before responding, but the receipt email, the analytics event and the warehouse update don’t. Moving that work out of the request is one of the most useful changes in a growing system.

Synchronous and asynchronous

In a synchronous call, the caller waits for the answer. This is simple, but it ties the two services together. If the receiver is slow, so is the caller. If it’s down, the caller fails too. As chapter 3 showed, dependencies in series multiply their failure rates.

In asynchronous communication, the caller leaves a message and moves on. A queue sits in between:

[Producer] --> [Queue] --> [Consumer]

This gives three benefits.

• Decoupling. The producer doesn’t need the consumer to be up.
• Load levelling. A spike fills the queue, and consumers work through it at a steady rate.
• Independent scaling. Consumers can be added as needed.

The cost is more moving parts, and a user may not get an immediate result. That usually means a design such as returning 202 Accepted and showing progress later.

Task queues

A task queue distributes work. RabbitMQ and Amazon SQS are common examples. A message is delivered to one worker, and when the worker acknowledges it, the message is removed.

If a worker crashes before acknowledging, the message becomes visible again after a timeout, and another worker takes it. This is why messages can be delivered more than once.

Task queues suit image resizing, sending email, and other jobs where each one needs doing once and order doesn’t much matter.

Log streams

A log stream, such as Apache Kafka or Amazon Kinesis, works differently. Messages are appended to an ordered log and aren’t deleted when read. They stay for a retention period, and each consumer tracks its own position, called an offset.

Partition 0:  [0][1][2][3][4][5][6]
                      ^           ^
                  Consumer B   Consumer A

This has some powerful consequences.

• Many independent consumers can read the same stream.
• Replay. A new consumer can start from the beginning, and a buggy one can rewind after a fix.
• Ordering is preserved within a partition.

Partitions and consumer groups

A topic is split into partitions so it can scale. Each message goes to a partition, usually by hashing a key, such as a user ID, so that all of one user’s events stay in order.

A consumer group is a set of consumers sharing the work. Each partition is read by one member of the group at a time. So the number of partitions limits a group’s parallelism, and ordering holds only within a partition, not across the whole topic.

Choosing a key has the same risks as sharding (chapter 9): one very active key can overload a partition.

Comparing them

Publish and subscribe

In pub/sub, a publisher sends to a topic, and every subscriber gets a copy. One order event can feed billing, shipping and analytics, and a new subscriber can be added without changing the publisher. Log streams provide this through consumer groups, and brokers offer it through fan-out.

The risk is hidden coupling. When nobody can see who depends on an event, changing it becomes dangerous. Schemas with versioning help, and a schema registry can check compatibility automatically.

When things go wrong

Poison messages

A message that always crashes its consumer will be retried forever, blocking others behind it. So after a few failures, messages go to a dead letter queue, a holding area for later inspection. A dead letter queue needs monitoring, because otherwise failed work piles up unseen.

Duplicates

Networks make duplicates unavoidable. A consumer can process a message and crash before acknowledging it, and a producer can time out after the broker stored its message. Chapter 13 covers this in depth. The short version is that consumers have to be idempotent, meaning that processing a message twice has the same effect as once.

Ordering

Retries break ordering. If message 2 fails and goes back in the queue, message 3 may be processed first. If order matters, the options are to process one key at a time, or to include a version number and ignore older updates.

Lag

Consumer lag is how far consumers are behind producers. It’s the key health metric for a stream. Rising lag means consumers can’t keep up. It’s better to alert on the age of the oldest message than on the queue’s length, because age relates directly to the delay users feel.

Backpressure

A queue absorbs bursts, but it doesn’t add capacity. If work arrives faster than it’s processed, on average, the queue grows without limit.

Backpressure is any way of telling a producer to slow down. TCP does this automatically with its receive window. In an application, a bounded queue that rejects or blocks when full provides the same signal.

An unbounded queue is dangerous. It hides overload, uses up memory, and makes every item wait longer. By the time an item is processed, the user may have given up, so the system is busy doing work nobody wants.

A bounded queue forces a choice about what happens when it’s full:

The choice depends on the data. For live metrics, the newest values matter most. For orders, nothing should be dropped silently.

Schedules and delays

Many systems need work done later, such as a reminder in a week or a retry in an hour. A simple way is a database table of due times, polled by a worker. Doing this safely across several workers takes care: SELECT ... FOR UPDATE SKIP LOCKED lets workers claim rows without blocking each other.

When not to use a queue

A queue adds operational work and makes flows harder to follow. If the user needs the answer now, a queue only adds delay. If a task is quick and rarely fails, a direct call may be better. For small systems, a database table often makes a perfectly good queue.

The takeaway

Queues separate producers from consumers, smooth bursts, and let parts of a system fail independently. They don’t add capacity, they do allow duplicates, and they need bounds. Decide what happens when a queue fills up before it happens.