27 — A notification system

A notification system takes an event inside a product, such as “someone commented on your post” or “your payment failed”, and turns it into a message on a phone, in an inbox, or as a text. Most of the work isn’t sending. It’s deciding whether to send, sending once, and coping with external providers that are slow, rate-limited and sometimes wrong about what happened.

Requirements

Functional

• Services raise notification events through one API.
• Three channels: mobile push (APNs for Apple devices, FCM for Android), email, and SMS.
• Per-user preferences: which categories on which channels, and quiet hours in the user’s time zone.
• Templates with variables and locales.
• Priorities: a password reset can’t wait behind a marketing campaign.
• Digests: twenty “new follower” events become one message.
• Delivery tracking: what was sent, when, and what each provider said.

Non-functional

• No silent loss. An accepted event is eventually sent, suppressed by a rule, or recorded as failed.
• No visible duplicates. Retries are unavoidable, so the design has to make them harmless.
• Latency by priority. Seconds for security messages. Minutes are fine for social ones, and hours for marketing.
• Provider limits are respected. Every provider throttles callers, and a sender that ignores this gets throttled harder.

Out of scope: in-app notification feeds (a read model like the feed in chapter 25), and building an email server.

Estimates

The method is from chapter 4. Every input below is an assumption, chosen to make the arithmetic concrete.

That average is misleading. A broadcast to 10 million devices in 10 minutes is:

So the peak is about 29 times the average, and it’s driven by one kind of traffic. This is the main reason channels and priorities get separate queues.

Storage for the log is $5 \times 10^7 \times 200\ \text{B} = 10\ \text{GB}$ a day, or about 900 GB for 90 days. That’s a partitioned PostgreSQL table, or a log store, but not a hard problem. SMS volume, at 2.5 million a day, matters mostly for cost, because carriers bill per segment.

API

One endpoint accepts events from other services:

POST /v1/notifications
Idempotency-Key: 7f1c9a52-orders-8812-shipped
Content-Type: application/json

{
  "user_id": 4821,
  "category": "order_shipped",
  "priority": "high",
  "template": "order_shipped_v3",
  "data": {"order_id": 8812, "carrier": "UPS"},
  "collapse_key": "order-8812"
}

HTTP/1.1 202 Accepted
{"notification_id": "ntf_01JC9V3T6R", "status": "queued"}

202 means accepted, not delivered. The caller can’t know about delivery yet, and shouldn’t wait for it. The Idempotency-Key makes a retried POST safe, as in chapter 15: a second request with the same key returns the first result.

Users manage settings through ordinary endpoints: GET and PUT /v1/users/{id}/preferences, plus device registration, POST /v1/users/{id}/devices, which stores a push token.

Data model

PostgreSQL holds the relational part: users’ devices, preferences and the notification records.

CREATE TABLE devices (
    device_id    bigserial PRIMARY KEY,
    user_id      bigint      NOT NULL,
    platform     text        NOT NULL CHECK (platform IN ('apns', 'fcm')),
    push_token   text        NOT NULL UNIQUE,
    last_seen_at timestamptz NOT NULL DEFAULT now(),
    disabled_at  timestamptz            -- set when the provider says the token is dead
);
CREATE INDEX ON devices (user_id) WHERE disabled_at IS NULL;

CREATE TABLE preferences (
    user_id     bigint  NOT NULL,
    category    text    NOT NULL,       -- 'order_shipped', 'new_follower', 'marketing', ...
    channel     text    NOT NULL,       -- 'push', 'email', 'sms'
    enabled     boolean NOT NULL,
    digest      text    NOT NULL DEFAULT 'none',   -- 'none', 'hourly', 'daily'
    PRIMARY KEY (user_id, category, channel)
);

CREATE TABLE user_settings (
    user_id      bigint PRIMARY KEY,
    time_zone    text   NOT NULL DEFAULT 'UTC',    -- an IANA name, e.g. 'Europe/Berlin'
    quiet_start  time,
    quiet_end    time,
    locale       text   NOT NULL DEFAULT 'en-US'
);

CREATE TABLE notifications (
    notification_id text        PRIMARY KEY,
    idempotency_key text        NOT NULL UNIQUE,
    user_id         bigint      NOT NULL,
    category        text        NOT NULL,
    priority        text        NOT NULL,
    payload         jsonb       NOT NULL,
    created_at      timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE deliveries (
    notification_id text        NOT NULL,
    channel         text        NOT NULL,
    target          text        NOT NULL,       -- device id, email address or phone number
    status          text        NOT NULL,       -- queued, sent, failed, suppressed, ...
    attempts        int         NOT NULL DEFAULT 0,
    provider_id     text,                       -- the provider's message id, if any
    last_error      text,
    created_at      timestamptz NOT NULL DEFAULT now(),
    updated_at      timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (notification_id, channel, target, created_at)
) PARTITION BY RANGE (created_at);

CREATE INDEX ON deliveries (provider_id);       -- callbacks look rows up by it

Some details matter here.

• The unique idempotency_key turns duplicate submissions into a constraint violation instead of a second message.
• The deliveries primary key names one attempt target. A worker that retries updates this row rather than inserting a new one.
• Partitioning deliveries by creation time lets old data go by dropping a partition, which is far cheaper than a large DELETE. PostgreSQL requires a partitioned table’s primary key to include the partition column, so created_at is part of the key. It never changes, so a row stays in one partition however many times it is updated.
• time_zone stores an IANA name, not an offset. An offset is wrong for half the year wherever daylight saving time applies.

Topology

 services ──> [Notification API] ──(one transaction)──> PostgreSQL: notifications + outbox
                                                               │
                                                       [outbox relay]
                                                               v
                                                   [Kafka: notification events]
                                                               │
                                        [Router: preferences, quiet hours, dedup, digest]
                                       ┌───────────────┬───────┴───────┬────────────────┐
                                       v               v               v                v
                                 push.high        push.normal       email            sms
                                       │               │               │                │
                                 [push workers]──> APNs / FCM   [email workers]  [sms workers]
                                       │                               │                │
                                       └──────── status updates ───────┴──> deliveries table

The API writes the notification and an outbox row in one transaction, and a relay publishes it. This is the transactional outbox from chapter 15. Without it, the API could commit the row and crash before publishing, and the event would be lost silently. The relay can publish twice after a crash, so everything downstream has to tolerate duplicates.

The router is the decision point. For each event it loads the user’s preferences and settings, applies the rules below, renders the template, and writes one job per channel and target to that channel’s queue. Workers then deal only with sending.

Deciding whether to send

The router runs these checks in order. Each one can end the process, and each outcome is recorded, because “why didn’t I get the email?” has to have an answer.

Security messages, such as password resets and login codes, are usually exempt from quiet hours and digests. Legal requirements also apply to some categories. For marketing email, RFC 8058 defines one-click unsubscribe: the message carries List-Unsubscribe-Post: List-Unsubscribe=One-Click and an HTTPS List-Unsubscribe URI, and both headers have to be covered by a DKIM signature.

Quiet hours

Quiet hours are stored in local time and checked in local time. The window may cross midnight, which is the usual case (22:00 to 08:00).

from datetime import datetime, time, timedelta, timezone
from zoneinfo import ZoneInfo


def next_send_time(now_utc, tz, start, end):
    # Return now if outside quiet hours, else the moment quiet hours end (in UTC).
    local = now_utc.astimezone(ZoneInfo(tz))
    t = local.time()
    quiet = (start <= t < end) if start < end else (t >= start or t < end)   # may cross midnight
    if not quiet:
        return now_utc
    day = local.date() if t < end else local.date() + timedelta(days=1)
    resume = datetime.combine(day, end, tzinfo=ZoneInfo(tz))
    return resume.astimezone(timezone.utc)


now = datetime(2026, 11, 9, 4, 30, tzinfo=timezone.utc)
for tz in ("America/New_York", "Europe/Berlin", "Asia/Kolkata", "Asia/Tokyo"):
    send = next_send_time(now, tz, time(22, 0), time(8, 0))
    local = now.astimezone(ZoneInfo(tz)).strftime("%H:%M")
    print(f"{tz:17} local {local}  ->  send at {send:%Y-%m-%d %H:%M} UTC")

Output:

America/New_York  local 23:30  ->  send at 2026-11-09 13:00 UTC
Europe/Berlin     local 05:30  ->  send at 2026-11-09 07:00 UTC
Asia/Kolkata      local 10:00  ->  send at 2026-11-09 04:30 UTC
Asia/Tokyo        local 13:30  ->  send at 2026-11-09 04:30 UTC

The same instant is late evening in New York and early afternoon in Tokyo. A delayed notification goes onto a delay queue, or into a table polled by a scheduler, keyed by its send time. One consequence for broadcasts is that “send to everyone at 9:00” really means 9:00 in each time zone, so a global broadcast becomes a rolling one over about a day. That also flattens the peak in the estimates.

Templates

A template is stored per category, channel and locale, with a version:

Rendering happens in the router, before queuing, for two reasons. The rendered text is what gets logged, so support can see exactly what was sent. And size limits can be checked once. Apple documents a 4 KB (4,096-byte) limit on a push payload (APNs docs). SMS is stricter: one GSM-7 segment holds 160 characters, but a message that needs UCS-2, for instance one containing an emoji or most non-Latin scripts, holds 70. Longer messages are split into segments of 153 or 67 characters, because each segment carries a header used to reassemble the message. Each segment is billed, so a template that accidentally includes one curly quote can double its SMS cost.

Templates should escape variables for their channel, HTML for email in particular. A user’s display name is user input, and putting it raw into an HTML email is an injection bug.

A queue per channel

Each channel gets its own queue, and push gets one per priority:

• Isolation. If the SMS provider is down, SMS jobs back up, while push and email carry on. One shared queue would block everything behind the failing jobs, the head-of-line blocking from chapter 8.
• Different rates. Each provider has its own rate limit and latency, so each channel needs its own worker count and pacing.
• Priority. Separate queues are the simplest reliable priority mechanism. Workers drain push.high first, or a fixed share of workers serves only high priority, so a 10-million-device campaign can’t delay a login code.

Kafka partitions by user ID keep one user’s notifications in order on a channel, which matters for pairs like “order shipped” and “order delivered”. Ordering across channels isn’t guaranteed and usually isn’t needed.

Deduplication

Duplicates come from several places, and each needs its own defense.

The fourth row is the hard one. A worker sends a push, the provider delivers it, and the HTTP response times out. The worker can’t tell this apart from a request that never arrived. Retrying risks a duplicate, and not retrying risks a loss. Providers offer a partial fix. APNs’ apns-collapse-id header merges notifications with the same value into one on the device, and FCM’s collapse key replaces an undelivered message with the newer one. Setting the collapse ID to the notification ID turns a duplicate send into a replacement rather than a second banner. Email and SMS have no equivalent at the device, so they get at-least-once delivery with a small chance of duplicates, or at-most-once with a small chance of loss. For a one-time login code, a duplicate is harmless. For “your payment failed”, it’s alarming but harmless. The choice is usually at-least-once.

Retries, backoff and provider rate limits

Provider responses fall into three groups, and each needs different handling.

FCM’s error documentation is explicit: for QUOTA_EXCEEDED (HTTP 429) it says to use exponential backoff with a minimum initial delay of one minute, and for UNAVAILABLE (503) to honor Retry-After and back off exponentially (FCM error codes). Apple’s guidance for APNs is to reuse each HTTP/2 connection as long as possible, often for hours or days, rather than opening one per notification. APNs’ own 429 is narrower than a global limit: it means too many requests to the same device token.

Retrying permanent errors wastes quota and can look like abuse to the provider, so dead tokens have to be cleaned up promptly.

Simulating a burst against a rate-limited provider

The simulation sends a burst of 10,000 notifications to a provider modeled as a token bucket at 100 requests a second. The provider’s behavior is an assumption, chosen to be unkind:

• 10% of accepted requests fail with a transient 503.
• 2% fail permanently, like an unregistered token.
• 3% are delivered, but the response is lost, so the sender sees a timeout.
• Requests over the rate get 429.
• Each notification gets at most 8 attempts.

Four senders are compared: a fixed 10 ms retry; exponential backoff with full jitter (base 0.5 s, cap 60 s, from chapter 16); backoff plus client-side pacing at the provider’s rate; and all of that plus a key the provider uses to merge repeats, which is how a collapse ID behaves.

import heapq
import random

PROVIDER_RATE = 100      # provider accepts 100 requests a second
PROVIDER_BURST = 100     # token bucket capacity at the provider
P_TRANSIENT = 0.10       # 503: provider overloaded, retry later
P_PERMANENT = 0.02       # 404: token unregistered, never retry
P_LOST_ACK = 0.03        # provider delivered, but the response timed out
MAX_ATTEMPTS = 8


class Provider:
    def __init__(self, rng):
        self.rng = rng
        self.tokens = PROVIDER_BURST
        self.last = 0.0
        self.seen = set()          # idempotency keys already delivered
        self.deliveries = {}       # notification id -> times shown to the user
        self.calls = 0
        self.rejected_429 = 0

    def send(self, now, nid, key):
        self.calls += 1
        self.tokens = min(PROVIDER_BURST, self.tokens + (now - self.last) * PROVIDER_RATE)
        self.last = now
        if self.tokens < 1:
            self.rejected_429 += 1
            return "429"
        self.tokens -= 1
        r = self.rng.random()
        if r < P_PERMANENT:
            return "404"
        if r < P_PERMANENT + P_TRANSIENT:
            return "503"
        if key is None or key not in self.seen:
            self.deliveries[nid] = self.deliveries.get(nid, 0) + 1
            if key is not None:
                self.seen.add(key)
        if r < P_PERMANENT + P_TRANSIENT + P_LOST_ACK:
            return "timeout"
        return "200"


def run(n, backoff, pace, idem, seed=7):
    rng = random.Random(seed)
    prov = Provider(random.Random(seed + 1))
    events = [(0.0, i, 1, False) for i in range(n)]  # (time, id, attempt, slotted): a burst at t=0
    heapq.heapify(events)
    next_slot = 0.0
    done_at, gave_up, dead = {}, 0, 0
    while events:
        now, nid, attempt, slotted = heapq.heappop(events)
        if pace and not slotted:  # client-side pacing: reserve the next send slot
            slot = max(now, next_slot)
            next_slot = slot + 1 / PROVIDER_RATE
            heapq.heappush(events, (slot, nid, attempt, True))
            continue
        res = prov.send(now, nid, f"n-{nid}" if idem else None)
        if res == "200":
            done_at[nid] = now
            continue
        if res == "404":
            dead += 1
            continue
        if attempt == MAX_ATTEMPTS:
            gave_up += 1
            continue
        if backoff:   # exponential backoff with full jitter, base 0.5 s, cap 60 s
            delay = rng.uniform(0, min(60.0, 0.5 * 2 ** (attempt - 1)))
        else:
            delay = 0.01   # fixed 10 ms retry
        heapq.heappush(events, (now + delay, nid, attempt + 1, False))
    lat = sorted(done_at.values())
    dup = sum(c - 1 for c in prov.deliveries.values() if c > 1)
    return {
        "calls": prov.calls, "429s": prov.rejected_429,
        "acked": len(done_at), "delivered": len(prov.deliveries),
        "duplicates": dup, "gave_up": gave_up, "dead": dead,
        "p50": lat[len(lat) // 2], "p99": lat[int(len(lat) * 0.99)], "last": lat[-1],
    }


if __name__ == "__main__":
    N = 10_000
    cases = [
        ("fixed 10 ms retry, no pacing, no key", False, False, False),
        ("backoff + jitter, no pacing, no key", True, False, False),
        ("backoff + jitter, paced, no key", True, True, False),
        ("backoff + jitter, paced, idempotency key", True, True, True),
    ]
    print(f"{'strategy':42} {'calls':>6} {'429s':>6} {'deliv':>6} {'dups':>5} {'404':>4} {'exhaust':>7} {'p50 s':>6} {'p99 s':>6}")
    for name, b, p, k in cases:
        r = run(N, b, p, k)
        print(f"{name:42} {r['calls']:>6} {r['429s']:>6} {r['delivered']:>6} {r['duplicates']:>5} "
              f"{r['dead']:>4} {r['gave_up']:>7} {r['p50']:>6.1f} {r['p99']:>6.1f}")

Output:

strategy                                    calls   429s  deliv  dups  404 exhaust  p50 s  p99 s
fixed 10 ms retry, no pacing, no key        79383  79276     90     2    4    9908    0.0    0.1
backoff + jitter, no pacing, no key         75670  70321   4676    44  112    5354   25.5   53.6
backoff + jitter, paced, no key             11541      0   9765   375  242       0   57.7  114.2
backoff + jitter, paced, idempotency key    11541      0   9765     0  242       0   57.7  114.2

deliv counts notifications that reached a user at least once, dups counts extra copies, 404 counts permanent failures, and exhaust counts notifications that ran out of attempts. The latency columns are the times of successful acknowledgments, from the start of the burst.

The results:

• Fixed fast retries delivered 90 of 10,000. Nearly 80,000 calls hit an empty bucket, and every notification burned its 8 attempts within about 80 ms.
• Backoff alone helped, but over half still ran out of attempts. Eight attempts with this backoff span roughly half a minute on average, and draining 10,000 messages at 100 a second takes at least 100 seconds. A retry budget shorter than the drain time can’t succeed, however well-jittered.
• Pacing fixed it. The sender never exceeded the provider’s rate, so there were no 429s, and the calls fell from 75,670 to 11,541, close to the minimum of 10,000 plus retries of real failures. Everything except the permanent failures was delivered, in about 114 seconds, which is the drain time.
• Lost acknowledgments caused 375 duplicates in the paced run. Merging repeats on a key removed all of them at the same call count.

Pacing turns a burst into a queue. Waiting in a queue is cheap, while waiting through 429s costs calls, quota and goodwill with the provider. The rate to pace at has to be shared among all workers sending to that provider, which makes it the distributed rate limiter from chapter 22, applied to outgoing traffic.

Digests and batching

Some events are worth bundling. Twenty “liked your photo” pushes in an hour are noise, while one “Ana and 19 others liked your photo” is useful.

A digest buffer is keyed by user, category and window:

CREATE TABLE digest_items (
    user_id      bigint      NOT NULL,
    category     text        NOT NULL,
    window_start timestamptz NOT NULL,
    event_id     text        NOT NULL,
    data         jsonb       NOT NULL,
    PRIMARY KEY (user_id, category, window_start, event_id)
);

A scheduler closes each window, renders one message from its items, and sends it through the normal path. The primary key makes inserting the same event twice harmless. Two variations are common:

• Leading edge. Send the first event at once, then batch the rest of the window. The first like is timely, and the next nineteen don’t spam.
• Collapse in place. Push channels can replace a notification already shown by reusing its collapse ID, so the count updates instead of stacking.

Batching on the sending side is a different thing. Email providers often accept many recipients per API call, which reduces request overhead. It doesn’t change what the user sees.

Delivery tracking

The deliveries row moves through states:

queued ──> sending ──> sent ──> delivered ──> opened
   │           │         │
   │           └──> retrying ──> failed
   └──> suppressed / delayed

sent means the provider accepted it. Later states come from callbacks, if at all. Email providers report bounces and complaints through webhooks, and SMS providers report carrier delivery receipts. Push providers generally don’t report delivery to the device, so “opened” comes from the app itself calling back.

Callbacks are at-least-once and can arrive out of order, so updates use the provider’s message ID and only move forward:

UPDATE deliveries
SET status = 'delivered', updated_at = now()
WHERE provider_id = $1
  AND status IN ('sent', 'retrying');   -- never move back from 'opened' or 'failed'

Hard bounces and complaints should disable the address. Continuing to email addresses that bounce damages the sender’s reputation with mailbox providers, and that reduces delivery for everyone else.

Failure modes

Two of these deserve emphasis. A notification can’t be unsent, so a bad campaign is worse than a bad web deploy. And queue age, the time the oldest job has waited, is the metric to alert on, because a queue can be short and stuck at once.

The takeaway

Sending is the easy part. The design is mostly about deciding whether to send, which needs preferences, local time and digests, and about making retries safe, which needs idempotency keys, collapse IDs and an outbox. Providers set the pace, so pacing to their rate does more good than any backoff scheme.