Distributed Transactions

User clicks "Buy." You must: charge card (payments service) + decrement inventory (warehouse service) + create order (orders DB). All three or none — partial is catastrophic (card charged, no order). In one database, ACID gives this for free. Across services, it doesn't exist.

The sobering fact: true distributed ACID is effectively impossible without paying huge latency and availability costs. Modern systems use three pragmatic alternatives: 2PC (avoid), Saga (use), and the Outbox pattern (the unsung hero).

Model the checkout as a sequence. Each step can fail; each step has a compensation that undoes it.

# 2PC (expensive, blocks on coordinator failure — prefer sagas where possible)

class Coordinator:
    def commit(self, txn_id, participants):
        # Phase 1: PREPARE
        votes = []
        for p in participants:
            try:
                votes.append(p.prepare(txn_id))  # writes UNDO + REDO log
            except Exception:
                votes.append(False)
        if not all(votes):
            for p in participants: p.rollback(txn_id)
            return "aborted"

        # Phase 2: COMMIT
        self.log.write(f"COMMIT {txn_id}")  # durable decision point
        for p in participants:
            while True:
                try: p.commit(txn_id); break
                except Exception: continue   # infinitely retry
        return "committed"

# Problem: coordinator crash between phase 1 and phase 2 blocks every
# participant holding prepared locks. This is why sagas + idempotency
# wins for most distributed-transaction use cases.

Two orchestration styles:

• Orchestrated saga — central orchestrator (e.g., state machine) tells each service what to do. Clear, debuggable, introduces a new service to own.
• Choreographed saga — each service listens to events and decides what to do. No central orchestrator. Simpler infra, harder to reason about.

Orchestrated wins for complex workflows (5+ steps). Choreographed wins for simple 2–3 step flows.

1. Blocking. All participants lock their resources until the coordinator says go/abort. One slow participant stalls the entire transaction. Under load, lock queues explode.
2. Coordinator SPOF. If the coordinator crashes after "commit" is sent to one participant but not others, the remaining participants don't know the outcome. They sit holding locks — indefinitely.
3. Unavailable during partitions. Network partition → coordinator or participant unreachable → entire transaction blocked. CAP CP behavior, but applied to every transaction across every service.

Message brokers (Kafka) and some DBs (Spanner via Paxos) offer transactional guarantees that feel like 2PC but with much better availability via consensus. Outside those narrow cases, don't roll your own 2PC.

The core problem with saga: when service A commits its local DB and needs to notify service B, if the notification fails, saga breaks. You cannot atomically "commit DB + send message" across systems (that's 2PC, which we're avoiding).

Outbox pattern: service A writes its DB change AND an "outbox" row in the same local transaction. A separate process polls outbox rows and publishes them to Kafka / a message bus. After successful publish, outbox row is marked sent (or deleted).

1. DB commit happens: both business data + outbox row are written atomically (one-DB transaction — free).
2. Poller sees new outbox rows, publishes to Kafka.
3. If poller crashes after publishing but before marking sent → republishes on restart → at-least-once delivery. Consumers must be idempotent.
4. If poller crashes before publishing → retries on restart → still at-least-once.

Net effect: atomicity between "change the DB" and "emit the event," without a distributed transaction. Every serious microservices system uses this (Debezium makes it turnkey via DB change-data-capture). Pair with idempotent consumers and sagas — this is the modern distributed transaction stack.

E-commerce checkout is a saga (payment + inventory + order + notification). Payment gateway uses outbox for webhook delivery. Ticketmaster uses saga + distributed locks to prevent double-booking. Bidding platform uses orchestrated sagas for auction settlement.