16 — Running it: observability, deployments and regions
A design isn’t finished when it’s built. Most of a system’s life is spent running, changing and occasionally breaking. Design choices decide whether those are routine or terrifying.
Observability
Observability is how well a system’s internal state can be understood from outside. Three signals work together.
| Signal | Answers | Strength | Weakness |
|---|---|---|---|
| Metrics | Is something wrong? | Cheap, good for alerts | No detail per request |
| Traces | Where is it wrong? | Follow one request across services | Usually sampled |
| Logs | Why is it wrong? | Full detail | Costly at volume |
Structured logs as key-value pairs can be searched, unlike free text. A correlation ID, passed through every service, ties one request’s logs together.
Distributed tracing records each step of a request as a span, with a timing and a parent. The result is a timeline across services, which makes a slow dependency obvious. OpenTelemetry is the common open standard.
High cardinality is a cost trap. A metric labelled by user ID creates a separate time series per user, which can overwhelm a metrics system.
What to measure
Two simple methods cover most needs.
The RED method applies to services:
- Rate: requests per second.
- Errors: failed requests.
- Duration: latency percentiles.
The USE method, from Brendan Gregg, applies to resources:
- Utilization: how busy it is.
- Saturation: how much work is queued.
- Errors: error events.
The SRE book’s four golden signals are similar: latency, traffic, errors and saturation.
Alerting
Alert on symptoms that affect users, not on every internal cause. High CPU alone isn’t a problem if users are fine. Burn-rate alerts fire when an error budget is being used too quickly. Every alert should need human action, because noisy alerts teach people to ignore them.
Deployments
Change is the leading cause of outages, so the aim is to make each change small, gradual and reversible.
| Strategy | How it works | Trade-off |
|---|---|---|
| Rolling | Replace instances a few at a time | Simple, with old and new mixed for a while |
| Blue-green | Run two environments, then switch traffic | Instant rollback, but double the capacity |
| Canary | Send a small share of traffic to the new version, then grow it | Limits blast radius, and needs good metrics |
| Shadow | Copy live traffic to the new version, and discard responses | Safe testing, but side effects need care |
A canary release might go 1%, 10%, 50%, 100%, comparing error rates and latency against the old version at each step, and rolling back automatically if they get worse.
Feature flags separate deployment from release. Code ships switched off, and it’s enabled for chosen users later. That allows gradual rollout and instant shutoff with no deployment. Old flags need removing, or they pile up as hidden complexity.
Compatibility. During a rollout, old and new versions run side by side, so they have to work with the same data and each other. This is why database changes use the expand-and-contract approach from chapter 5.
Configuration is code. Bad config changes have caused many of the largest outages, so they deserve the same gradual rollout as code.
Infrastructure
Containers package an application with its dependencies. Kubernetes schedules them onto machines, restarts failed ones and handles networking. Infrastructure as code, using tools such as Terraform, makes environments reproducible and reviewable.
Multi-region
Running in several regions protects against a whole-region failure, and reduces latency for distant users. It’s expensive and hard, so it needs a clear reason.
| Pattern | Recovery | Cost |
|---|---|---|
| Backup and restore | Hours | Low |
| Warm standby | Minutes | Medium |
| Active-active | Seconds | High |
Two numbers define recovery targets:
- RPO, the recovery point objective: how much data can be lost.
- RTO, the recovery time objective: how long recovery can take.
A key test is whether failover has been practised. An untested standby often fails when it’s needed, because of forgotten config or capacity limits. Regular failover drills are the only real proof.
Blast radius matters here too. Cell-based architecture splits a system into isolated copies, each serving a slice of users, so one failure or bad deployment hits only that cell.
Dependencies you don’t own
Third-party services fail as well. Critical ones need timeouts, fallbacks, and ideally a second provider. Static stability is a useful principle: a system should keep working if its control plane fails, so running servers shouldn’t need a management service in order to carry on.
Incidents
When things break, a clear process helps.
- Mitigate first. Roll back or shed load before finding the root cause.
- Assign roles. One person leads, others investigate.
- Communicate with regular updates.
- Write a blameless postmortem. Focus on how the system allowed the failure.
Runbooks record known fixes. Roll back is usually the fastest fix, which is why deployment rollback has to be simple.
Cost
Cost is a design constraint too. Cross-region and cross-zone traffic, log volume and over-provisioning are common surprises. Measuring cost per request makes it visible.
The takeaway
Metrics say something’s wrong, traces say where, and logs say why. Deploy small changes gradually, and make rollback easy. Practise failover, because an untested plan isn’t a plan.