14 — Resilience: timeouts, retries, circuit breakers and overload

Large systems are always partly broken. The aim of resilience is to stop a small failure from growing into a large one. The usual worst case is a cascading failure, and the patterns here are defences against it.

How a cascade happens

Most serious outages aren’t caused by one component dying. They follow a pattern like this:

1. A database slows down.
2. Service B, which calls it, has every thread stuck waiting.
3. Service A, which calls B, runs out of threads too.
4. Health checks start failing, and servers are removed from rotation.
5. The remaining servers get more load, and fall over as well.

A slow dependency is often worse than a dead one. A dead dependency fails instantly, while a slow one ties up resources for as long as it’s allowed to. The Google SRE book names overload as the most common cause of cascading failures. It also describes a nasty property: a service that crashed at 11,000 queries a second may not recover at 9,000.

Timeouts

Every network call needs a timeout. Without one, a hung dependency holds a thread forever, and many libraries have no default timeout, or a very long one.

Choosing a value takes some care. Too long, and threads pile up. Too short, and slow-but-working calls fail. A common starting point is slightly above the dependency’s p99.

Deadline propagation improves on this. If a user request has 2 seconds in total, and 1.5 have gone, then the next service should get 500 ms, not a fresh 2 seconds. gRPC supports this directly. Without it, work carries on long after the user has gone.

Retries

Many failures are brief, so a retry often works. But retries are dangerous under overload, because they add load at exactly the wrong moment.

Retries multiply across layers. If each layer tries three times:

So one failing request can become hundreds. Retrying at only one layer avoids that.

Backoff and jitter

Exponential backoff waits longer after each failure: 100 ms, 200, 400, and so on. Jitter adds randomness, so that clients don’t all retry at the same moment.

A simulation had 100 clients all fail at once, against a server that handles 10 requests per 100 ms, with a maximum backoff of 10 seconds:

Without jitter, the clients stayed in step and kept colliding. Full jitter, which waits a random time between zero and the backoff limit, spreads them out. The Amazon Builders’ Library gives a detailed account of this.

As the Google SRE book puts it: “If at first you don’t succeed, back off exponentially”, and always add jitter.

Retry budgets

A retry budget caps retries as a share of normal traffic, say 10%. When a dependency is healthy, retries are rare and the budget doesn’t matter. When it’s failing, retries stop, instead of multiplying the load.

Only safe operations should be retried: those that are idempotent (chapter 13), and only on errors that might go away. A 400 Bad Request won’t change on retry, and a 503 might.

Circuit breakers

If a dependency is clearly failing, continuing to call it wastes threads and adds to its load. A circuit breaker, a pattern named in Michael Nygard’s Release It!, stops the calls, the same way an electrical breaker cuts power.

It has three states:

              failure rate too high
   +--------+ -------------------> +------+
   | CLOSED |                      | OPEN |
   +--------+ <--+                 +------+
                 |                    |
          trial  |                    | wait time passes
        succeeds |                    v
                 |              +-----------+
                 +------------- | HALF-OPEN |
                                +-----------+
                  trial fails: back to OPEN

• Closed. Calls pass through, and results are recorded.
• Open. Calls fail immediately, with no network call.
• Half-open. After a wait, a few trial calls test whether the dependency has recovered.

Libraries such as Resilience4j use a sliding window of recent calls, and open when the failure rate passes a threshold, 50% by default, once a minimum number of calls has been made. That minimum stops a single early failure tripping the breaker.

Here’s a small implementation:

import time
from collections import deque

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    def __init__(self, window=20, min_calls=10, failure_rate=0.5, open_seconds=5.0):
        self.results = deque(maxlen=window)     # True = success
        self.min_calls = min_calls
        self.failure_rate = failure_rate
        self.open_seconds = open_seconds
        self.state = "CLOSED"
        self.opened_at = 0.0

    def call(self, fn):
        if self.state == "OPEN":
            if time.monotonic() - self.opened_at < self.open_seconds:
                raise CircuitOpen()             # fail fast, no network call
            self.state = "HALF_OPEN"            # let one trial call through
        try:
            result = fn()
        except Exception:
            self._record(False)
            raise
        self._record(True)
        return result

    def _record(self, ok):
        if self.state == "HALF_OPEN":
            self.results.clear()
            if ok:
                self.state = "CLOSED"
            else:
                self._trip()
            return
        self.results.append(ok)
        calls = len(self.results)
        failures = calls - sum(self.results)
        if calls >= self.min_calls and failures / calls >= self.failure_rate:
            self._trip()

    def _trip(self):
        self.state = "OPEN"
        self.opened_at = time.monotonic()
        self.results.clear()

if __name__ == "__main__":
    healthy = {"up": True}

    def dependency():
        if not healthy["up"]:
            raise TimeoutError("dependency timed out")
        return "ok"

    breaker = CircuitBreaker(window=10, min_calls=5, open_seconds=0.2)
    counts = {"ok": 0, "failed": 0, "failed_fast": 0}

    def run(n):
        for _ in range(n):
            try:
                breaker.call(dependency)
                counts["ok"] += 1
            except CircuitOpen:
                counts["failed_fast"] += 1
            except TimeoutError:
                counts["failed"] += 1

    run(20)
    print("healthy:", counts, breaker.state)
    healthy["up"] = False
    run(100)
    print("outage: ", counts, breaker.state)
    healthy["up"] = True
    time.sleep(0.25)
    run(20)
    print("after:  ", counts, breaker.state)

Output:

healthy: {'ok': 20, 'failed': 0, 'failed_fast': 0} CLOSED
outage:  {'ok': 20, 'failed': 5, 'failed_fast': 95} OPEN
after:   {'ok': 40, 'failed': 5, 'failed_fast': 95} CLOSED

During the outage, 5 calls failed before the breaker opened, and the other 95 failed instantly, using no resources. After the wait, a trial call succeeded, and the breaker closed.

What it does for a system

A simulation sent 100 requests a second for 60 seconds, to a dependency that fails for 30 seconds. Failed calls hang for a 2-second timeout, and the caller has 50 threads:

Successes are about the same, because the dependency was down either way. The difference is in the other 3,000 requests. Without a breaker, the 50 threads spend the outage stuck in timeouts. If those threads also served other work, it would all stop too. With a breaker, threads stay free, and users get an instant error.

Fallbacks

A breaker needs a plan for what happens when it’s open. Options include:

Falling back to something sensible is called graceful degradation. Fallback paths need testing, because they run only when things are already going wrong.

Tuning

• Breakers need one per dependency, and sometimes one per endpoint.
• A window that’s too small flips on noise.
• With many clients, recovery can cause a stampede, so half-open traffic needs limiting.

Bulkheads

Ships are divided into watertight sections, so one breach doesn’t sink them. A bulkhead does the same for resources: each dependency gets its own thread pool or connection limit. If the recommendations service hangs, it can use up its own pool, while checkout carries on.

Backpressure and load shedding

Chapter 8 showed that queues near capacity turn into latency. Load shedding rejects work early, so the rest can succeed.

Without shedding, an overloaded server does worse than expected. It accepts everything, each request times out, and it spends its time on work nobody is waiting for. Throughput of useful work can fall towards zero.

Good load shedding has several parts:

• Reject early and cheaply, before doing any work.
• Prioritize. Tag requests by importance, so that a checkout outranks a background sync. The SRE book calls this criticality.
• Respect deadlines. Drop requests that have already expired.
• Limit concurrency. Cap requests in flight. Adaptive concurrency finds the limit by watching latency.

Clients can help. In adaptive throttling, a client tracks how often its requests are accepted, and starts refusing requests locally as the backend refuses more, which spares the backend from processing them.

Metastable failures

There’s a particularly nasty kind of failure, where a system stays broken after the original cause has gone. Bronson et al. call these metastable failures.

A trigger pushes the system into a bad state, and a feedback loop keeps it there. Retries are the classic loop. A simple model has a server with capacity 100 a second and normal load of 80, with a five-second spike to 150:

Without retries, it recovers by itself. With them, the spike is long gone, and the system stays down, because retried requests hold the load above capacity. This model is crude, but the shape matches real incidents. It explains why removing the original trigger doesn’t help, and why recovery often requires cutting load sharply, then letting it back in gradually.

Cold caches create the same trap: a restart empties the cache, the database is overloaded, so requests time out, and the cache never fills.

Testing failure

Resilience features that have never been exercised often fail when needed.

• Load tests find the actual limits, and show how a system behaves past them.
• Fault injection adds latency and errors in controlled ways.
• Chaos engineering, which started with Netflix’s Chaos Monkey, deliberately kills things in production to prove that the system copes.
• Game days rehearse incidents with the people who’d handle them.

How they fit together

For one call, a typical stack is:

Deadline check -> Load shedder -> Bulkhead -> Circuit breaker
    -> Timeout -> Retry with jitter and budget -> Network call

Each answers a different question.

Order matters. A retry inside a circuit breaker counts differently from one outside it, so the choice should be deliberate.

The takeaway

Slow failures spread, so every call needs a timeout. Retries have to be limited, jittered and budgeted, or they can keep an outage going. Circuit breakers and bulkheads stop one dependency taking everything down, and load shedding keeps a system useful when it’s overwhelmed. All of this needs testing before it’s needed.