Delivery Guarantees

You send a message. The network is flaky. Did it arrive? If you retry, does the consumer see it twice? If you don't retry, did it get lost?

Three answers exist: at-most-once (may drop), at-least-once (may duplicate), exactly-once (the elusive unicorn). Every message system's correctness story starts with which one it gives you. Every consumer's correctness story starts with "I assume at-least-once, so I'm idempotent."

The difference boils down to when the consumer acknowledges:

• ACK before processing → at-most-once. Consumer receives message, acks immediately, then processes. If consumer crashes mid-process, message is already acked (gone from queue) — lost.
• ACK after processing → at-least-once. Consumer receives, processes, then acks. If consumer crashes before acking, message stays in queue → redelivered on retry → duplicated.

Everyone picks at-least-once. The solution to duplicates is not a stronger guarantee — it's idempotent consumers.

Exactly-once across independent systems (producer → queue → consumer → downstream) is provably impossible without transactional coordination. The proof: generals problem. Any message to the downstream can be lost; any response can be lost. You always have some retry strategy, which means either "don't retry" (at-most-once) or "retry until confirmed" (at-least-once).

But Kafka claims exactly-once! True — within Kafka, with the transactional API. It works because:

1. Producer and broker share a transaction ID; the broker deduplicates messages with the same producer + sequence number.
2. Consumer reads messages AND commits offset in the same Kafka transaction. Read + offset-commit succeed together or both get rolled back.
3. "Exactly once from Kafka topic → Kafka topic" is the only rigorous claim. Writing to an external DB at the end (side effect) breaks the guarantee unless the external DB is also part of the transaction.

Idempotent consumers are the real answer. Assign each message a unique ID. The consumer maintains a "seen" set (usually a DB unique constraint). Duplicate message → unique constraint violation → ignore silently. Logically exactly-once with at-least-once delivery underneath.

1. Deterministic side effects. If processing the same message twice produces the same result, you don't need anything special. "Set user X status to active" is idempotent. "Increment user X counter" is not.
2. Unique-key upserts. INSERT ... ON CONFLICT DO NOTHING. The second delivery of the same message becomes a no-op.
3. Processed-ID table. Maintain processed_events(event_id). Start of handler: INSERT; on duplicate-key error → already processed, return. On success, commit business change + INSERT together in one DB transaction.
4. Idempotency keys in APIs. Client sends Idempotency-Key header; server stores first response by key; retries return the cached response. Stripe's approach.

Payment gateway requires idempotency keys for every charge. WhatsApp message delivery uses at-least-once with client-side deduplication. Notification system uses at-least-once with idempotent hooks. Distributed queue exposes at-least-once semantics by default.