23 — Approaching a design

The chapters before this one cover parts. This one covers how to put them together for a new problem. The chapters after it apply the process to specific systems: a URL shortener, a news feed and a chat system, with more after those.

A design is a set of decisions, each made for a reason that can be written down. The process below puts those decisions in an order where each one has the information it needs.

The process

The steps are in order, but the process loops. A deep dive often changes the data model, and a failure case often changes the topology.

1. Requirements

Requirements come in two kinds.

Functional requirements are the operations, written as verbs: create a short link, follow a redirect, post, read a feed, send a message. Three to five is usually enough. Anything else is listed as out of scope, so that it doesn’t quietly grow the design.

Non-functional requirements say how well each operation must work:

These qualities are rarely uniform. A redirect must be fast and highly available, and the analytics behind it can be minutes late. A chat message must not be lost, and a typing indicator can be. Writing the qualities per operation, rather than for the system as a whole, is what makes the later trade-offs possible.

2. Estimates

Estimates turn the requirements into numbers that decide the design: average and peak requests a second, the read-to-write ratio, storage growth, bandwidth and the size of the data that’s read often. Chapter 4 covers the method.

Every input is an assumption, and should be labeled as one. The point is the order of magnitude. The useful question after each number is whether it fits on one machine (chapter 2). Chapter 6 works out, for example, that a pool of 20 connections running 10 ms queries tops out at 2,000 queries a second. An estimate well under a limit like that keeps the design simple. One far above it means splitting the work.

Two results matter more than the rest:

• The read-to-write ratio. A system that’s read 100 times for every write wants caches and precomputed answers. A write-heavy system wants cheap appends and work deferred off the write path.
• The multipliers. One user action that causes many writes (a post fanned out to followers) or many reads (a page that calls twenty services) changes the real load far more than the user count does.

3. API

The API lists the main operations with their inputs and outputs. It fixes the boundaries of the system before any boxes are drawn. Some details are worth settling here, because they affect everything behind them (chapter 17):

• Idempotency. Which operations can be retried safely, and which need a client-supplied key (chapter 15).
• Pagination. Lists need cursors rather than offsets once data changes while someone pages through it. Chapter 25 shows why.
• Errors. What a caller sees when it’s throttled, when an object is missing, and when the service is overloaded.
• Synchronous or not. Whether the caller waits for the work, or gets an acknowledgement and the result later.

4. Data model, from access patterns

The data model starts from the queries, not from the entities. For each operation, the question is which data it reads or writes, by which key, how often, and how fresh it must be.

PostgreSQL is a sound default for anything relational until a number from step 2 says otherwise (chapter 6). When the data outgrows one machine, the partition key is chosen here, from the access patterns: it should keep each common query on one partition, and spread load evenly (chapter 11).

5. Topology

The topology is the set of components and the path each main request takes through them. A small diagram is enough:

[Client] -> [Load balancer] -> [Stateless service] -> [Cache] -> [Database]
                                       |
                                       v
                                 [Queue] -> [Workers]

The defaults are stateless services behind a load balancer (chapter 5), caches in front of read bottlenecks (chapter 7), and queues in front of slow or bursty work (chapter 8). Each component in the diagram should be there because a number or a requirement put it there.

Tracing the top two or three operations through the diagram, step by step, catches missing pieces: a lookup with no index, a write that has to reach two stores, a request that waits on work that could be deferred.

6. Deep dives

Most of a design is standard. One or two parts aren’t, and they deserve most of the time. The next section is about finding them.

7. Failure

For each component, the questions are what happens when it’s down, when it’s ten times slower than usual, and when it returns wrong or stale data. The answers usually involve timeouts, retries with backoff and jitter, circuit breakers, fallbacks and load shedding (chapter 16), and a decision about which operations to keep working and which to give up.

Overload deserves its own pass (chapter 10): what happens at twice the expected peak, and what happens when every client reconnects at once after an outage.

Finding the hard part

A design is hard where a simple, standard answer stops working. Those places tend to show up as one of a few signals.

Three questions help find them.

1. Which estimate is largest relative to one machine? Dividing each number by what one server can do (from chapter 2 and measurement, chapter 3) shows which part needs to be distributed. The rest can often stay simple.
2. What happens at ten times the load? A part that degrades smoothly is fine. A part where one user’s action costs ten times more, or where one key gets ten times hotter, is the hard part.
3. What must be exactly right? Duplicate payments, lost messages and messages out of order are worse than a slow page. Correctness requirements across machines are almost always a deep dive.

A deep dive then works like a smaller design: list the options, estimate their cost with the numbers from step 2, pick one, and say what it gives up. A simulation or a small program is often the fastest way to settle a question that would otherwise be argued. The design chapters that follow use one for each of their main decisions.

Patterns that recur

The same handful of ideas solve the hard parts of most designs.

Two of these deserve a note.

Source of truth versus derived data. Most designs have one store that must not lose data, and several that are derived from it: caches, feeds, search indexes, counters. Knowing which is which settles a lot of failure questions. Derived data can be rebuilt, so it can live in memory and be dropped under pressure. The source of truth needs replication and backups (chapter 12).

Hot keys. Every large system has a few keys far more popular than the rest. A design that treats all keys alike will be sized for the average and broken by the maximum. The usual answers are to cache the hot keys closer to readers, to replicate them, or to handle them on a different path. All three appear in the chapters that follow.

Writing it down

A design that exists only as a diagram leaves out the reasons. A short written design keeps them. A workable outline:

1. Problem       One paragraph. Who uses it and why.
2. Requirements  Operations, qualities per operation, out of scope.
3. Estimates     Each number, with its assumption.
4. API           Main operations.
5. Data model    Tables or keys, and the query each serves.
6. Topology      A diagram, and the path of the top operations.
7. Decisions     For each hard part: options, choice, what it gives up.
8. Failure       Each dependency down or slow, and the response.
9. Operations    Metrics, alerts, rollout and rollback.
10. Open issues  What's not decided yet.

The decisions section matters most. A reader a year later needs to know why fan-out stops at a given follower count, not only that it does.

A design checklist

1. Numbers. Read-to-write ratio, data volume, latency targets, peak load and the main multipliers.
2. Data. Access patterns, storage engine, partition key and which store is the source of truth.
3. Topology. Stateless compute, with caches before read bottlenecks and queues before slow writes.
4. Hot keys. What happens to the most popular key, account or conversation.
5. Consistency. Which data needs strong guarantees, and which can be stale, and for how long.
6. Ordering and uniqueness. Where they’re required, and who assigns them.
7. Idempotency. Which operations are retried, and how duplicates are detected.
8. Failure. What happens when each dependency is down, or ten times slower.
9. Overload. Timeouts, retry budgets, circuit breakers, load shedding and reconnect storms.
10. Code boundaries. Domain logic separated from storage and transport (chapter 21).
11. Concurrency. Where state is shared, and who owns it.
12. Security. Authentication, authorization, abuse and data exposure (chapter 19).
13. Operations. Metrics, alerts, rollback and capacity (chapter 20).

The takeaway

A design is a sequence of decisions driven by a few numbers. Most of it is standard, and the work is in finding the one or two parts where the standard answer breaks: a multiplier, a hot key, an ordering or uniqueness rule, or long-lived state. The simplest design that meets the real requirements is usually the right one, and writing down why each decision was made is part of the design.