Time Synchronization & Clocks

Two machines, separated by a network, disagree on the order of events. "Event A at 12:00:00.500" on server 1. "Event B at 12:00:00.400" on server 2. Did B come before A? Only if the clocks agree, and they never perfectly do. Clocks drift, jump, lag, and sometimes go backwards. Any system that reasons about "what happened first" is building on unstable ground unless it's explicit about which clock, and why.

Time is the stealth cause of most distributed-systems bugs. Understanding the distinction between physical and logical clocks, and which to use when, is foundational.

Every node maintains a counter t.

• Local event → t = t + 1.
• Send a message → include current t.
• Receive a message with timestamp t' → t = max(t, t') + 1.

Result: if event A happened-before event B (causally), then A's timestamp < B's timestamp. The converse isn't true — smaller-numbered doesn't prove causal precedence, just potential precedence. Lamport timestamps give you a total order; vector clocks give you a partial order with precise causal comparisons.

Google Spanner wanted globally-consistent transactions — strict serial order of commits across the planet. Logical clocks alone don't provide this (they don't connect to real time). NTP is too loose (100ms is an eternity).

Google built TrueTime: every datacenter has GPS receivers AND atomic clocks, redundantly. Every machine queries a local time master which reports [earliest, latest] — a bounded uncertainty interval. Typical bound: ε ≈ 5ms worldwide, guaranteed.

Spanner uses TrueTime to implement commit wait: before acknowledging a write at timestamp T, wait until local TrueTime's "earliest" exceeds T. Ensures that any observer polling after the ack sees T in their past. Provides external consistency — a globally-linearizable order of commits — at the cost of 5–10ms extra latency per commit.

Nobody else has this because nobody else wants to install atomic clocks in every datacenter. CockroachDB mimics it with Hybrid Logical Clocks (NTP + logical counter) — almost-as-good external consistency without the atomic-clock CapEx.

• For timeouts — use monotonic clock, never wall clock. Wall-clock jumps break timeouts; monotonic ones don't.
• For audit logs — wall clock + NTP is fine. Humans will read them; 100ms error doesn't matter.
• For ordering distributed events — logical clocks (Lamport / vector). If you need wall-time approximation, use HLC.
• For distributed transactions — TrueTime if you're Google, HLC otherwise. Don't roll your own.
• For leases + expiry — include enough margin for clock skew. "Lease expires at T" should be treated as "definitely expired by T+ε" by any consumer.

Google Docs uses logical clocks (OT sequence numbers) for concurrent edits. Stock exchange uses PTP for matching-engine determinism. Distributed logging uses HLC for log ordering. Payment gateway uses monotonic clocks for idempotency-key expiry.