3 — Measuring: latency, throughput, availability

Design decisions need numbers. Without them, “fast”, “scalable” and “reliable” don’t mean anything testable. This chapter covers the handful of measurements that most decisions rest on.

Latency and throughput

Latency is how long one operation takes. Throughput is how many operations finish per unit of time.

A road is a useful picture. Latency is how long one car takes to get from A to B. Throughput is how many cars pass a point each minute. Adding lanes raises throughput without making any one car faster. Raising the speed limit improves latency.

The two can pull against each other. Batching is the classic case: collecting events and writing them together raises throughput, because one write carries many events. But each event now waits for its batch. Flushing every 50 ms adds up to 50 ms of waiting, and a simulation with uniform arrivals gave a mean of 25.0 ms.

Why averages mislead

If nine requests take 10 ms and one takes 1,000 ms, the average is 109 ms, which describes none of them.

Percentiles give a truer picture. The p50 is the median: half of requests are faster. The p99 is the value that 99% of requests beat.

The slow end matters more than it first appears, for two reasons.

Heavy users hit it more. A user making many requests is likely to see the p99 often. These may well be the most valuable users.

Pages make many requests. If a page makes 50 requests, the chance of at least one being slower than the p99 is $1 - 0.99^{50} \approx 39\%$. So the p99 of a backend can easily be the typical experience of a page. Chapter 8 returns to this.

A common mistake: percentiles can’t be averaged. The average of each server’s p99 isn’t the fleet’s p99. Accurate fleet-wide figures come from combining histograms.

Availability

Availability is the share of time, or of requests, that a system works. It’s usually quoted in “nines”.

Each extra nine is ten times harder. At five nines, there isn’t time for a person to notice a problem and respond, so recovery has to be automatic.

Series and parallel

Two simple rules explain most of availability design.

Dependencies in series multiply. If a request needs five services, each at 99.9%, the result is $0.999^5 \approx 99.5\%$. That’s worse than any one service. Adding required dependencies always lowers availability.

Redundancy in parallel multiplies failure. Two independent copies at 99.9% are both down only $0.001^2$ of the time, which gives 99.9999%.

The catch is the word “independent”. Copies that share a power supply, a network, a configuration or a software bug can fail together. In practice, shared causes like bad deployments and configuration changes are a major source of outages, so real redundancy gains are smaller than the arithmetic suggests.

SLIs, SLOs and error budgets

The Google SRE book gives a useful vocabulary.

• A service level indicator (SLI) is a measurement, such as the share of requests that succeed within 300 ms.
• A service level objective (SLO) is a target for it, such as 99.9% over 30 days.
• A service level agreement (SLA) is a contract, with consequences if it’s missed.

An SLO implies an error budget. A 99.9% SLO allows 0.1% of requests to fail. At 100 million requests a month, that’s 100,000 failures.

The budget turns reliability into a decision. If there’s budget left, a team can ship faster and take risks. If it’s used up, the focus moves to stability. It also shows why 100% is the wrong target: it would mean never changing anything.

Estimating capacity

Rough estimates, often called back-of-the-envelope calculations, show whether a design is one machine or a hundred. Precision doesn’t matter here, only the order of magnitude.

Some handy numbers:

Worked example

A service has 100 million requests a day. 1% are writes of 1 KB, and reads return 10 KB.

Now consider the servers. If each handles 500 requests a second comfortably, that’s about 7 at peak, plus spares. The data fits easily on one database for years. This is a modest system, and sharding it would add cost and complexity for nothing.

The peak-to-average ratio is an assumption, and it varies by product. Traffic follows daily patterns, and events can cause much sharper spikes, so real measurements should replace the guess as soon as they exist.

Storage has hidden multipliers

Raw data size is rarely the full story.

Headroom

Capacity planning needs spare room for failures. If a system runs on three servers, and one fails, the other two take its load. So each should run below about 66% at peak to survive that. Running across three zones gives the same result at zone level. This is called N+1 or N+2 planning.

The takeaway

Use percentiles rather than averages. Treat availability as arithmetic over dependencies. Do rough capacity calculations before designing anything, because they often show that the hard problem isn’t where it seemed.