13 — Delivery guarantees and transactions across services

Once data lives in several systems, the simple atomic transaction is gone. This chapter covers how systems stay correct without one.

Delivery guarantees

The reason is the two generals problem. If a sender gets no acknowledgement, it can’t tell whether the message or the acknowledgement was lost. So it either retries and risks a duplicate, or doesn’t and risks a loss. Most systems choose at-least-once, and handle duplicates.

Idempotency

An operation is idempotent if doing it several times has the same effect as once.

The standard technique for a consumer records each event ID in the same transaction as the work:

BEGIN;
INSERT INTO processed_events (event_id) VALUES ('evt-123')
  ON CONFLICT DO NOTHING;
-- if a row was inserted, do the work
UPDATE accounts SET balance = balance + 10 WHERE id = 7;
COMMIT;

The check and the work commit together, so a retry finds the event ID and does nothing.

Idempotency keys in APIs

A payment API can’t simply retry a timed-out request, because the charge may have gone through. So a client generates a unique idempotency key and sends it with the request. The server stores the key with the result, and a retry with the same key gets the original result rather than a second charge. Payment APIs such as Stripe’s work this way.

Keys need to be stored atomically with the operation, kept for a defined time, and tied to the request body, so that reusing a key with different data is an error.

The dual write problem

This looks innocent:

db.save(order)
broker.publish("order_created", order)

If the process crashes between the two lines, the order exists and no event was sent. Swapping the lines gives the opposite problem: an event for an order that doesn’t exist. Two systems can’t be updated atomically without a shared transaction.

Transactional outbox

The outbox pattern solves this by using one transaction. The event goes into an outbox table in the same database as the data:

BEGIN;
INSERT INTO orders (id, total) VALUES (42, 100);
INSERT INTO outbox (topic, payload) VALUES ('order_created', '{"id": 42}');
COMMIT;

A separate process reads the outbox and publishes each event, marking it as sent. Either both rows exist or neither does. If the publisher crashes after publishing, and before marking, the event goes out twice, so consumers still need to be idempotent. The pattern gives at-least-once delivery, with nothing lost.

Change data capture

Databases already record every change in their write-ahead log. Change data capture (CDC) reads that log and turns it into a stream of events. Debezium is a widely used tool for this, and it can read an outbox table from the log with no polling.

CDC has many uses:

• Keeping search indexes and caches up to date.
• Feeding analytics.
• Migrating data between systems with little downtime.

Sagas

A business process that spans services can’t use one transaction. A saga splits it into local transactions, each with a compensating action that undoes it.

A trip booking is a typical example:

If step 3 fails, the saga runs the compensations for steps 2 and 1, in reverse.

Sagas have two coordination styles.

Choreography has each service react to events from the others. There’s no central part, but the overall flow is hard to see, and cycles can creep in.

Orchestration has a central coordinator tell each service what to do. The flow is explicit and easy to monitor, but the coordinator is one more service to run. Workflow engines such as Temporal are built for this.

Sagas differ from transactions in an important way. There’s no isolation: other users can see intermediate states, such as a flight reserved for a trip that later fails. Some effects also can’t be undone. A sent email can only be followed by an apology. Ordering steps so that the hardest-to-undo comes last helps a lot.

What exactly-once really means

Kafka offers exactly-once semantics, and it’s real, within its boundary. The idempotent producer attaches sequence numbers, so the broker discards duplicates from retries. Transactions make a set of writes across partitions, including consumer offsets, atomic. So a read-process-write pipeline between Kafka topics can act as if each message was processed once.

Outside that boundary, it doesn’t apply. If a consumer calls an external API or writes to a separate database, a crash can still repeat that effect. So “exactly once” in practice means at-least-once delivery plus idempotent processing.

Event sourcing

Most systems store current state. Event sourcing stores the sequence of events instead, such as Deposited 100, Withdrew 30, and works out state by replaying them. A bank ledger is a natural example.

It gives a full audit history, the ability to rebuild state, and a way to create new views from old events. The costs are real too: schemas have to evolve while old events stay forever, replays can be slow without snapshots, and deleting personal data is hard.

CQRS, command query responsibility segregation, separates the write model from read models. The write side enforces rules, and read models are built from events, each shaped for one screen. It pairs well with event sourcing, and it means reads are eventually consistent.

Reconciliation

Even with all of this, systems drift. Bugs and manual fixes happen. Many high-value systems run a regular reconciliation job that compares two systems and reports differences. Payment systems compare their records with a bank’s daily. It’s a plain approach, and it often catches what cleverness misses.

The takeaway

Assume at-least-once delivery, and make operations idempotent. Never rely on two separate writes both succeeding: use an outbox or CDC. Use sagas for multi-service processes, and design their compensations from the start. “Exactly once” holds within a boundary, and past that it’s idempotency that does the work.