2 — What one machine can do

Distributed systems are expensive and hard to run, so the first design question is whether one machine would do. A surprising number of systems never need more than one good server and a replica. Answering that takes a rough model of what a machine does.

The four resources

Every server has four resources, and a system is limited by whichever runs out first.

Speed differs by huge factors

The gaps between storage layers shape almost every design.

These are rough figures that drift with hardware, but the ratios are what matter. Memory is about a thousand times faster than an SSD, so keeping the right data in memory is one of the most effective changes available.

Processes and threads

A process is a running program, with its own private memory. A thread is a line of execution inside a process, and threads in one process share memory.

There are three common ways to serve many users at once.

One process or thread per connection. It’s simple, and each request’s code reads top to bottom. But each thread costs memory and scheduling work, so tens of thousands of connections become expensive.

A thread pool. A fixed number of threads handle requests from a queue. This bounds resource use, which is a useful property. If all threads are busy, new work waits.

An event loop. One thread handles many connections by never blocking. It starts an operation, moves on, and returns when the operation finishes. Node.js and Nginx work this way. It’s efficient for network-heavy work, but one slow CPU task stalls everything on that loop.

Languages such as Go offer lightweight threads, managed by the runtime, which give the simple code of the first model with costs nearer the third.

Blocking and CPU-bound work

Two kinds of work behave very differently.

I/O-bound work spends its time waiting for a database or another service. A thread that’s waiting uses almost no CPU, so one machine can handle many such requests at once.

CPU-bound work actually computes, as in image resizing or encryption. It can only run on as many requests at once as there are cores.

Most web services are I/O-bound, which is why they can often handle far more concurrent requests than they have cores.

How much can one server do?

Rough numbers help here, though they vary a great deal with the work involved.

A rough model divides capacity by the work per request. With 16 cores and requests that use 10 ms of CPU each:

$\frac{16 \text{ cores} \times 1000 \text{ ms}}{10 \text{ ms}} = 1{,}600 \text{ requests/s}$

That’s the ceiling at 100% CPU. As chapter 8 shows, running near that is a bad idea, so a sensible target is well below it.

The same calculation with 1 ms of CPU per request gives 16,000 a second. So efficient code directly reduces the machines needed.

Databases have a similar shape. A database connection pool of 20 connections with 10 ms queries handles at most:

$\frac{20}{0.01 \text{ s}} = 2{,}000 \text{ queries/s}$

Vertical scaling

Moving to a bigger machine is called vertical scaling or scaling up. It has real advantages:

• No code changes.
• No new failure modes, because there’s still no network between parts of the system.
• Strong consistency stays simple, because there’s one copy of the data.

Its limits are the largest available machine, rising cost at the high end, and the fact that a bigger machine is still a single point of failure.

Modern servers are very large. Machines with hundreds of cores and terabytes of memory are ordinary cloud offerings, so the point where a single database stops being enough arrives much later than many teams expect.

What a process loses on restart

Anything held only in memory disappears when a process restarts. That includes sessions, in-memory queues and caches. Designing for this early helps later, because the same discipline makes it possible to run several copies of a process.

Durability

Data is only durable once it’s on stable storage. Writing to a file usually puts data in the operating system’s cache first, and it isn’t safe from a power cut until an fsync call has finished.

Databases use a write-ahead log to manage this. A change is appended to a log and flushed to disk before it’s acknowledged, and the main data files can be updated later. After a crash, the log is replayed. Because appending is sequential, it’s fast, and this is one of the most common ideas in storage systems.

Batching several commits into one flush, called group commit, is what lets a database do far more commits per second than a disk can do flushes.

The takeaway

Start by asking what one machine can do, and do the arithmetic. Many systems that look like they need a distributed design fit on one large server with a replica. The cost of distribution is high, and the next chapters show why it should be a deliberate choice.