20 — Running it: observability, deployments and regions

A design isn’t finished when it’s built. Most of a system’s life is spent running, changing and occasionally breaking. Design choices decide whether those are routine or terrifying.

Observability

Observability is how well a system’s internal state can be understood from outside. Three signals work together.

OpenTelemetry is the common open standard for producing all three, with libraries for most languages and a collector that forwards data to whichever backends a team uses.

Structured logs

Structured logs are key-value records, usually one JSON object per line, rather than free text. They can be filtered and aggregated by field, which free text can’t.

{"ts":"2026-11-09T10:00:02.513Z","level":"error","service":"payments","msg":"charge failed",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7",
 "order_id":"order-512","provider":"cardco","http_status":502,"duration_ms":3012,"attempt":2}

A correlation ID, passed through every service, ties one request’s logs together. When tracing is in place, the trace ID does this job, and putting it in every log line lets a trace and its logs be joined.

Log levels only help if they mean the same thing everywhere:

Two rules keep logs useful and safe. First, an error logged at every layer it passes through becomes five log lines for one failure, so log it once, where it’s handled. Second, logs are copied widely and kept for a long time, so they must never contain passwords, tokens, secrets or unnecessary personal data. Redaction belongs in the logging library, not in each call site.

Volume is the cost. As an assumption, take a service at 2,000 requests a second writing one 1 KB log line per request. That’s 2 MB a second, or 172.8 GB a day, before replication and indexing. Common ways to cut it:

• Sample successful requests, for example 1%, and keep every error and every slow request. At 1%, the example drops to about 1.7 GB a day of request logs.
• Make sampling consistent with tracing, by deciding from the trace ID, so a sampled request has both its logs and its trace.
• Use metrics for counting. A log line per request to count requests is an expensive counter.
• Keep debug logging switchable at runtime for one service or one user, instead of always on.

Metrics and cardinality

A metric is a named number over time, with labels that split it, such as http_requests_total{route="/orders", status="500"}. Each distinct combination of label values is a separate time series that the metrics system stores and indexes.

Series multiply. A latency histogram with 12 buckets has 14 series per label combination, counting its sum and count. With 5 methods, 40 routes and 5 status classes, that’s 1,000 combinations and 14,000 series per instance, or 700,000 across 50 instances. Adding a user_id label with 100,000 users multiplies that by 100,000, which no metrics system will handle well. This is high cardinality, and it’s the most common way to overload a metrics backend.

The Prometheus instrumentation guide suggests keeping the cardinality of most metrics below 10, limiting the few that exceed it, and investigating alternatives for anything over 100 or likely to grow that large. Unbounded values such as user IDs, request IDs, email addresses and full URLs belong in logs and traces, which are built for per-request detail. Route templates such as /orders/{id} are fine as labels; raw paths such as /orders/8812 aren’t.

What to measure

Two simple methods cover most needs.

The RED method applies to services:

• Rate: requests per second.
• Errors: failed requests.
• Duration: latency percentiles.

The USE method, from Brendan Gregg, applies to resources:

• Utilization: how busy it is.
• Saturation: how much work is queued.
• Errors: error events.

The SRE book’s four golden signals are similar: latency, traffic, errors and saturation.

The methods divide the work. RED, measured at every service boundary, says which service is unhealthy as its users see it. USE, applied to that service’s CPUs, memory, disks, connection pools and queues, points at the resource behind it. Saturation is often the earliest warning, because a queue grows before latency does (chapter 10).

Histograms, not averages

Chapter 3 explained why percentiles describe latency better than averages, and noted that percentiles can’t be averaged. The mistake is easy to make in a dashboard: each server reports its own p99, and the dashboard shows their mean.

A simulation makes the size of the error concrete. Nine healthy servers each handle 90,000 requests with a median around 20 ms. A tenth server is slow, with a median around 200 ms, and handles 10,000.

import bisect, random

random.seed(7)

def p(values, q):
    s = sorted(values)
    return s[min(len(s) - 1, int(q * len(s)))]

# Ten servers. Nine are healthy and busy; one is slow and gets little traffic.
servers = []
for i in range(9):
    servers.append([random.lognormvariate(3.0, 0.4) for _ in range(90_000)])  # ~20 ms median
servers.append([random.lognormvariate(5.3, 0.4) for _ in range(10_000)])      # ~200 ms median

per_server = [p(s, 0.99) for s in servers]
everything = [x for s in servers for x in s]
print("per-server p99 (ms):", [round(x) for x in per_server])
print(f"average of p99s:  {sum(per_server) / len(per_server):7.1f} ms")
print(f"max of p99s:      {max(per_server):7.1f} ms")
print(f"true fleet p99:   {p(everything, 0.99):7.1f} ms")
print(f"fleet mean:       {sum(everything) / len(everything):7.1f} ms")

# Histograms: each server counts requests into shared bucket bounds. Counts add up.
bounds = [5, 10, 25, 50, 100, 150, 200, 250, 300, 400, 500, 750, 1000, float("inf")]
merged = [0] * len(bounds)
for s in servers:
    for x in s:
        merged[bisect.bisect_left(bounds, x)] += 1
target, seen = 0.99 * len(everything), 0
for i, c in enumerate(merged):
    if seen + c >= target:
        lo = bounds[i - 1] if i else 0
        est = lo + (bounds[i] - lo) * (target - seen) / c   # linear interpolation in the bucket
        print(f"p99 from merged histogram: {est:.1f} ms (bucket {lo}-{bounds[i]} ms)")
        break
    seen += c

per-server p99 (ms): [51, 51, 51, 51, 51, 50, 51, 51, 51, 509]
average of p99s:     96.7 ms
max of p99s:        509.2 ms
true fleet p99:     139.8 ms
fleet mean:          24.2 ms
p99 from merged histogram: 137.0 ms (bucket 100-150 ms)

Histograms work because bucket counts can be added across servers and time windows, and a percentile can be estimated from the sum. The Prometheus documentation on histograms and summaries makes the same distinction: a summary computes quantiles on each instance, and those can’t be aggregated; a histogram exposes bucket counts, which can. The accuracy depends on bucket boundaries, so boundaries should be dense around the SLO threshold. Here a 100 to 150 ms bucket gave an estimate within 3 ms.

SLOs and burn-rate alerts

Chapter 3 defined SLIs, SLOs and error budgets. A 99.9% availability SLO over 30 days allows 0.1% of requests to fail. The burn rate is how fast the budget is being spent, relative to the rate that would use it up exactly at the end of the window. A burn rate of 1 means an error rate of 0.1%. A burn rate of 14.4 means 1.44%, which would empty a 30-day budget in about 2 days.

Simple alerts on error rate behave badly. Alerting whenever the 5-minute error rate exceeds 0.1% fires on blips that don’t threaten the SLO. Alerting on a long window detects real problems slowly and keeps firing long after they end.

The Google SRE workbook chapter on alerting on SLOs recommends multiwindow, multi-burn-rate alerts. Each alert fires only when the burn rate exceeds its threshold over both a long window, which shows the problem is significant, and a short window, which shows it’s still happening. Its suggested starting values:

The simulation below applies those three alerts, and a naive “5-minute error rate above 0.1%” alert, to five incidents, with constant traffic and one data point per minute.

SLO_ERR = 0.001
BUDGET_MIN = 30 * 24 * 60          # minutes in the SLO window
ALERTS = [  # name, long window, short window (minutes), burn-rate threshold
    ("page 1h/5m x14.4", 60, 5, 14.4),
    ("page 6h/30m x6", 360, 30, 6.0),
    ("ticket 3d/6h x1", 4320, 360, 1.0),
]

def run(name, error_ratio, start, length, total=6 * 24 * 60):
    errs = [error_ratio if start <= t < start + length else 0.0 for t in range(total)]
    prefix = [0.0]
    for e in errs:
        prefix.append(prefix[-1] + e)
    def avg(t, w):                       # mean error ratio over the last w minutes ending at t
        lo = max(0, t + 1 - w)
        return (prefix[t + 1] - prefix[lo]) / w
    used = sum(errs) / (SLO_ERR * BUDGET_MIN)
    print(f"\n{name}: budget used {used:.1%}")
    for label, long_w, short_w, rate in ALERTS:
        th = rate * SLO_ERR
        firing = [avg(t, long_w) > th and avg(t, short_w) > th for t in range(total)]
        if not any(firing):
            print(f"  {label:18} never fires")
            continue
        first = firing.index(True)
        last = total - 1 - firing[::-1].index(True)
        print(f"  {label:18} fires {first - start:5d} min after start, "
              f"clears {last + 1 - (start + length):5d} min after it ends")
    naive = [avg(t, 5) > SLO_ERR for t in range(total)]
    print(f"  {'naive 5m > 0.1%':18} firing for {sum(naive)} minutes")

start = 24 * 60
run("100% errors for 10 minutes", 1.00, start, 10)
run("2% errors for 3 hours", 0.02, start, 180)
run("0.5% errors for 2 days", 0.005, start, 2 * 24 * 60)
run("0.15% errors for 4 days", 0.0015, start, 4 * 24 * 60)
run("5% errors for 5 minutes", 0.05, start, 5)

The results, in minutes after the incident starts (fires) and after it ends (clears):

The pattern matches the workbook’s reasoning. A full outage pages immediately. A moderate error rate pages once it has used about 2% of the budget, which took 43 minutes at 2%. Slow burns, which threaten the month but not the next hour, open a ticket instead of waking someone. A short blip that used 0.6% of the budget alerts nobody. The short window is what makes alerts clear soon after recovery: the 1-hour page cleared 4 minutes after the outage ended, where a 1-hour window alone would keep firing for an hour. The naive alert, by contrast, fired for days on the slow burns and for the harmless blip.

When several alerts fire for one incident, the alerting system should group or suppress them so one problem produces one page.

Distributed tracing and context propagation

Distributed tracing records each step of a request as a span, with a start time, a duration, attributes and a parent span. Spans that share a trace ID form a tree, which shows where the time went across services. A slow dependency is obvious in it.

For this to work, every service has to pass the trace context to the next one. The W3C Trace Context recommendation standardizes the HTTP header for it:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  trace-id (16 bytes, hex)         parent-id        trace-flags (01 = sampled)
             version                             (8 bytes, hex)

A second header, tracestate, carries vendor-specific data alongside it. On receiving a request, a service keeps the trace ID, creates a new span whose parent is the incoming parent-id, and sends its own span ID as the parent-id on outgoing calls. OpenTelemetry’s libraries do this automatically for common HTTP and RPC frameworks.

The common gaps are anywhere the automatic propagation doesn’t reach:

• Message queues. The producer has to write the trace context into message headers, and the consumer has to read it. Chapter 18 puts it in every event envelope.
• Thread pools and async tasks. Work handed to another thread loses the context unless it’s passed explicitly.
• Batch consumers, where one span processes messages from many traces. Span links, which reference other traces without being their parent, model this.
• Third-party services that drop unknown headers.

Trace sampling keeps the cost bounded. OpenTelemetry’s sampling documentation describes two kinds.

Head sampling records the decision in the trace-flags field so every service agrees on it, and whole traces are kept or dropped. Tail sampling is what makes “keep every error and every trace slower than 1 second, plus 1% of the rest” possible, at the price of running a stateful pipeline.

Alerting

Alert on symptoms that affect users, not on every internal cause. High CPU alone isn’t a problem if users are fine. Burn-rate alerts, described above, fire when an error budget is being used too quickly. Every alert should need human action, because noisy alerts teach people to ignore them.

Alert fatigue

Alert fatigue is what happens when people receive more alerts than they can act on: they start acknowledging pages without investigating, and the real one gets lost. It builds up gradually, one reasonable-looking alert at a time.

Practices that keep it in check:

• Page only for urgent, user-visible problems that need a human now. Everything else becomes a ticket or a dashboard.
• Review every page afterwards. Was it real? Was action needed? Could it have waited? An alert that is repeatedly “no action needed” gets fixed or deleted.
• Track page volume per on-call shift as a health metric of the alerting itself.
• Group related alerts, so a database failure produces one page, not one per dependent service.
• Give every alert an owner. An alert nobody owns is never tuned.
• Don’t page on causes such as “disk 80% full” when a symptom-based alert, or a ticket with a long lead time, would serve.

Runbooks

A runbook is the document an alert links to. It’s written for the person who gets paged at 3 a.m. and has never seen this alert before.

A useful runbook contains:

1. What the alert means in one or two sentences, and what users are probably seeing.
2. How to confirm it: the dashboard, query or command that shows the problem.
3. Immediate mitigations, in order of safety: roll back the last deployment, shift traffic, shed load, scale up, fail over.
4. How to tell whether a mitigation worked.
5. Whom to escalate to, and when.
6. Links to the service’s architecture, dependencies and past incidents.

Runbooks rot as systems change, so they’re best reviewed after each incident that used them. A step that is the same every time is a candidate for automation, and a runbook that says “restart it” every time points at a bug to fix.

Deployments

Change is the leading cause of outages, so the aim is to make each change small, gradual and reversible.

A canary release might go 1%, 10%, 50%, 100%, comparing error rates and latency against the old version at each step, and rolling back automatically if they get worse.

Feature flags separate deployment from release. Code ships switched off, and it’s enabled for chosen users later. That allows gradual rollout and instant shutoff with no deployment. Old flags need removing, or they pile up as hidden complexity.

Compatibility. During a rollout, old and new versions run side by side, so they have to work with the same data and each other. This is why database changes use the expand-and-contract approach from chapter 6.

Configuration is code. Bad config changes have caused many of the largest outages, so they deserve the same gradual rollout as code.

Infrastructure

Containers package an application with its dependencies. Kubernetes schedules them onto machines, restarts failed ones and handles networking. Infrastructure as code, using tools such as Terraform, makes environments reproducible and reviewable.

Multi-region

Running in several regions protects against a whole-region failure, and reduces latency for distant users. It’s expensive and hard, so it needs a clear reason.

Two numbers define recovery targets:

• RPO, the recovery point objective: how much data can be lost.
• RTO, the recovery time objective: how long recovery can take.

A key test is whether failover has been practiced. An untested standby often fails when it’s needed, because of forgotten config or capacity limits. Regular failover drills are the only real proof.

Blast radius matters here too. Cell-based architecture splits a system into isolated copies, each serving a slice of users, so one failure or bad deployment hits only that cell.

Dependencies you don’t own

Third-party services fail as well. Critical ones need timeouts, fallbacks, and ideally a second provider. Static stability is a useful principle: a system should keep working if its control plane fails, so running servers shouldn’t need a management service in order to carry on.

Incidents

When things break, a clear process helps.

1. Mitigate first. Roll back or shed load before finding the root cause.
2. Assign roles. One person leads, others investigate.
3. Communicate with regular updates.
4. Write a blameless postmortem. Focus on how the system allowed the failure.

Runbooks, described above, record known fixes. Rollback is usually the fastest fix, which is why deployment rollback has to be simple.

Cost

Cost is a design constraint too. Cross-region and cross-zone traffic, log volume, metric cardinality and over-provisioning are common surprises. Measuring cost per request makes it visible.

The takeaway

Metrics say something’s wrong, traces say where, and logs say why. Keep metric labels bounded, aggregate latency with histograms, and sample logs and traces deliberately, keeping errors. Alert on SLO burn rate over a long and a short window, page only for what needs a human now, and link every alert to a runbook. Deploy small changes gradually, and make rollback easy. Practice failover, because an untested plan isn’t a plan.