4 — Scaling out: stateless servers and load balancers
When one machine isn’t enough, or when its failure would take the site down, the next step is several machines. This is horizontal scaling, or scaling out. The application tier is usually the easiest place to start.
The stateless idea
Running two copies of an application is only simple if they’re interchangeable. The obstacle is state: data that one server holds and another doesn’t.
The classic example is a login session. If server A keeps a user’s session in memory, that user has to keep reaching server A. Load balancers can do that with sticky sessions, but they bring problems:
- Load spreads unevenly.
- A restart logs people out.
- Removing a server for maintenance disrupts its users.
The fix is to move state out of the application servers and into a shared store. Sessions can go in Redis or a database, and files can go in object storage. Each server then holds nothing that matters between requests, which makes it stateless.
+--> [App 1] --+
[Client] --> [LB] ---+--> [App 2] --+--> [Redis] [Database]
+--> [App 3] --+Once servers are stateless, any of them can handle any request. Adding capacity means adding servers, and a failed server costs only its in-flight requests.
Load balancers
A load balancer sits in front of the servers and spreads requests across them. It also removes failed servers from rotation and gives clients one stable address.
Layer 4 and layer 7
The names come from the network model.
Layer 4 balancers work at the connection level, forwarding TCP or UDP without reading the contents. They’re fast and simple, but they can only balance connections, not individual requests.
Layer 7 balancers understand HTTP. They can route by path, retry failed requests, terminate TLS and balance individual requests. They cost more CPU per request.
Many systems use both: layer 4 at the edge, then layer 7 in front of services.
Choosing a server
| Algorithm | How it works | Weakness |
|---|---|---|
| Round robin | Each server in turn | Ignores differences in load |
| Weighted round robin | Bigger servers get more | Weights need maintaining |
| Least connections | Fewest active connections | Needs current load |
| Hashing | Same key goes to the same server | Hot keys overload one server |
| Random | Pick at random | Uneven under load |
| Power of two choices | Pick two at random, use the less loaded | Needs load information |
The last one is a striking result. A simulation placed 100,000 requests on 100,000 servers:
| Choices per request | Busiest server’s load |
|---|---|
| 1 (random) | 7 |
| 2 | 4 |
| 3 | 3 |
Comparing just two random servers does most of the work. It also avoids a trap in least connections: with several balancers working from stale data, all of them can send traffic to the same apparently idle server at once. This is known as the power of two random choices.
Health checks
A balancer has to know which servers are healthy.
A liveness check asks whether a process is running. A readiness check asks whether it can serve traffic right now. A server that’s starting up, or that’s lost its database connection, can be alive but not ready.
There are two common failure modes. A check that’s too sensitive can remove healthy servers during a brief slowdown, which pushes more load onto the rest and makes things worse. And if a shared dependency such as a database fails, a check that depends on it can mark every server unhealthy at once. Many balancers respond by “failing open”: if everything looks unhealthy, they send traffic to all servers anyway.
Graceful shutdown
During a deployment, a server should stop accepting new requests, finish those in flight, and only then exit. This is called connection draining, and without it, each deployment drops requests.
Who balances the balancer?
A load balancer is itself a single point of failure unless it’s redundant. Common approaches are a pair with a shared floating address, or anycast, where many sites announce the same IP address and the network routes users to the nearest.
DNS and global traffic
DNS can return different addresses for the same name, which gives a coarse kind of balancing. It’s useful for directing users to a nearby region. But DNS results are cached, so changes aren’t instant, and a client can keep using a dead address until its cache expires. So DNS alone is a slow way to route around failure.
Content delivery networks
A CDN is a network of servers around the world that caches content close to users. A request from Singapore can be answered in Singapore, without travelling to a server in Virginia.
CDNs help in three ways. They cut latency, because round trips are shorter. They reduce load on the origin, sometimes by a very large share. And they absorb large traffic spikes and some attacks.
Static content like images and scripts is the obvious fit. Dynamic content can also be cached briefly. Even caching a popular page for a few seconds means the origin regenerates it once rather than thousands of times.
Cache-busting is the usual way to update static files: including a content hash in the filename, like app.3f2a9c.js, lets a file be cached for a very long time, since any change produces a new name.
Autoscaling
Autoscaling adds and removes servers based on load, usually CPU or request rate. It suits predictable daily patterns well. It has limits, though.
New servers take time to start, sometimes minutes, so autoscaling can’t absorb a sudden spike. And if the real bottleneck is a database, adding application servers just adds more pressure on it.
What moves to the data tier
Scaling the application tier is the easy part, because it pushes the hard problem elsewhere. All those stateless servers now share a cache and a database, and this is where the difficult work starts. The next chapters are mostly about that shared state.
The takeaway
Make servers stateless, and scaling the application tier becomes routine. Put a load balancer in front, give it good health checks, and drain connections on shutdown. Then accept that the difficulty has moved to the data layer.