System Design — 016

Bidding Platform

Design an eBay-scale auction system where sellers list items, buyers place real-time bids, proxy bidding resolves automatically, and auctions close with sub-second precision — all while guaranteeing financial correctness under extreme concurrency.

Linearizable WritesReal-Time Fan-OutRedis LuaCQRSFinancial Correctness
01

Problem Statement

Design a platform supporting millions of simultaneous timed auctions. Sellers list items with a starting price, optional reserve, and end time. Buyers place bids in real-time. The system supports proxy bidding (automatic counter-bids up to a secret maximum) and anti-sniping (time extensions when bids arrive near the deadline).

The core tension: the system must be both blindingly fast (sub-200ms bid acknowledgment) and absolutely correct (the wrong person must never win an auction). In most distributed systems you pick availability and tolerate stale reads. Here, for the bid-write path, you need linearizable consistency — and you need it at low latency.

Core question: How do you serialize millions of concurrent bids across 50M auctions — especially when a single hot auction receives 50+ bids/second in its final seconds — without sacrificing speed, correctness, or durability?

02

Requirements

Functional Requirements

  • Auction lifecycle: Sellers create listings with start price, reserve price (hidden), bid increment, and end time. States: DRAFT → SCHEDULED → ACTIVE → CLOSING → CLOSED → SETTLED.
  • Bid placement: Buyers place bids that must exceed current highest + minimum increment. Response is immediate and unambiguous: "You are the highest bidder" or "You've been outbid."
  • Proxy bidding: Buyer sets a secret max. System auto-bids the minimum needed. Two competing proxies resolve instantly via a mathematical Vickrey-like formula — no iterative loop.
  • Anti-sniping: Configurable per-auction. Bids in the final N minutes extend the auction by N more minutes, up to a max number of extensions.
  • Real-time updates: All watchers see price changes within 1-2 seconds via WebSocket push.
  • Auction closure: Precise to within 1 second. In-flight bids accepted via a 2-second grace window using API Gateway timestamps.
  • Settlement: Payment hold → shipment → delivery confirmation → capture → seller payout. Full dispute and chargeback handling.
  • Audit trail: Immutable, append-only bid log with hash-chained event records for regulatory compliance.

Non-Functional Requirements

  • Linearizable consistency on the bid-write path — no stale reads, no lost updates, no double-acceptance.
  • Sub-200ms bid acknowledgment latency end-to-end.
  • 99.99% availability for bid placement (~52 min downtime/year).
  • Handle 100x traffic spikes during auction sniping surges.
  • Auction closure within 1 second of scheduled end time.
  • Zero bid loss — acknowledged bids must survive server crashes.
  • Complete auditability for dispute resolution and regulatory compliance.
03

Scale Estimation

Derived from eBay-scale assumptions: 150M registered users, 50M MAU, 15M DAU. Only ~10% of DAU are active bidders.

50M
Active auctions
~50/s
Bids/sec (avg)
50/s
Peak bids on 1 auction
25 GB
Hot auction state (all)
500K+/s
Peak push messages
300 GB/yr
Bid storage

Key Derivation

500K auctions closing/day × 8 avg bids = 4M bids/day → ~50/sec average. The critical number is the per-auction peak: a hot auction with 50 bids in 5 seconds = 10-50 bids/sec on a single Redis key. Each Lua script executes in ~0.5ms, so 50 bids/sec consumes only 25ms of Redis CPU — well within capacity. The real bottleneck is fan-out: 50 bids/sec × 50K watchers = 2.5M push messages/sec for one auction.

04

API Design

Place a Bid (Critical Path)
POST /v1/auctions/{auction_id}/bids
Authorization: Bearer <token>
Idempotency-Key: "bid_8f14e45f"

{ "amount": 79000, "type": "MANUAL" }

→ 200 OK
{
  "bid_id": "bid_Qm3nR8xP",
  "status": "ACCEPTED",
  "outcome": "OUTBID",          // proxy countered instantly
  "your_bid": 79000,
  "current_price": 79500,       // proxy's counter-bid
  "bid_count": 26,
  "anti_snipe": { "extensions_used": 1, "exhausted": false }
}

→ 409 Conflict  (bid too low — concurrency signal, not validation error)
→ 503 Retry     (auction migrating or Redis failover)
WebSocket — Real-Time Updates (Read-Only Push)
CONNECT wss://ws.bidplatform.com/v1/auctions/{auction_id}

← { "type": "BID_UPDATE", "current_price": 79500,
    "bid_count": 26, "your_status": "OUTBID",
    "time_remaining_seconds": 258, "reserve_met": false }

← { "type": "TIME_EXTENDED", "new_end_time": "...",
    "extension_number": 1, "max_extensions": 3 }

← { "type": "AUCTION_CLOSED", "final_price": 95000,
    "winner": "u***K", "you_won": true }

Design principle: REST for actions (idempotent, structured errors), WebSocket for observations (real-time, read-only). Bids are never placed via WebSocket — no idempotency guarantees on fire-and-forget channels. The Idempotency-Key is mandatory — sniping users will retry frantically on timeouts.

05

High-Level Architecture

Seven layers, each with a clear responsibility. The critical insight: the bid-write path is synchronous and in-memory (Redis Lua), while everything else is async via Kafka. This split — strong consistency on writes, eventual consistency on reads — is the fundamental architectural decision.

Clients Web / Mobile API Gateway Auth · Rate Limit Bid Service Stateless · Coalesce Auction Engine Redis Lua Scripts Serialization Point Kafka Event Backbone PostgreSQL Durable Store WebSocket Tier Pub/Sub Fan-Out Scheduler Timing Wheel Notifications Push · Email · SMS Elasticsearch Search Index Fraud Engine ML · Fingerprint Settlement Escrow · Payout HTTPS REST Lua Script Publish Pub/Sub Consume Async Write Close Cmd
Request Flow — Step Through
ClientAPI GatewayBid ServiceRedis LuaKafkaWebSocket TierWatchers
Click Next Step to walk through the request flow.
06

Deep Dive — Auction Engine Serialization

The Redis Lua script is the heart of the system. Every bid — manual or proxy — is processed atomically in a single Lua execution. Redis's single-threaded model guarantees linearizability by construction: no locks, no retries, no race conditions. The script validates, accepts, resolves proxies, checks anti-snipe, and returns the outcome in ~0.5ms.

Why Naive Approaches Fail

PostgreSQL Optimistic

UPDATE WHERE current_price = expected. Under contention, retry storms: bid N+1 retries after N succeeds. At 50 bids/sec, latency explodes.

SELECT FOR UPDATE

Row lock held for 5-20ms (network + fsync). 50th bid waits 500ms in queue. App crash holds lock for 30 seconds.

The Lua Script Pipeline

The script executes in 7 steps, all within a single atomic Redis call:

1. Read state — HGETALL auction state (~0.1ms). 2. Validate — auction active, not expired, not seller, not already highest, amount ≥ min. 3. Accept bid — update price, highest bidder, increment count. 4. Resolve proxies — mathematical resolution: winner pays min(loser_max + increment, winner_max). No iterative loop. 5. Anti-snipe — if within window and extensions remaining, extend end_time. 6. Write state — HMSET updated fields (~0.1ms). 7. Return — JSON result with outcome, price, flags.

Proxy Resolution — The Vickrey Formula

When two proxies compete (Alice max $500, Bob max $350), the result is computed in one step: Alice wins at min($350 + increment, $500) = $360. No looping through 35 counter-bids. Tie-breaking uses a composite score encoding max × 10¹³ + (10¹³ − timestamp), so the earlier proxy wins ties.

sequenceDiagram participant C as Client participant GW as API Gateway participant BS as Bid Service participant R as Redis (Lua) participant K as Kafka participant WS as WebSocket Tier C->>GW: POST /bids (amount=$500) GW->>GW: Auth + Rate Limit GW->>BS: Forward + X-Received-At BS->>BS: Idempotency check BS->>R: EVALSHA bid_processor.lua R->>R: Validate → Accept → Proxy resolve → Anti-snipe R-->>BS: {ok:true, outcome:OUTBID, price:510} BS-->>C: 200 OK (45ms total) BS->>K: Publish BID_ACCEPTED K->>WS: Consumer reads event WS->>WS: Personalize per-connection WS-->>C: Push: BID_UPDATE (80ms)

Critical Constraint: Script Must Complete in <2ms

Redis is single-threaded. While the Lua script runs, ALL other commands on that shard are queued — including bids for other auctions. At 0.5ms average with 50 ops/sec on a hot key, we consume only 25ms/sec of Redis time. The sorted set operations (proxy lookup) are O(log N) due to skip list internals. Monitoring: p99 script time > 1ms triggers an alert; > 2ms triggers investigation.

07

Key Design Decisions & Tradeoffs

Bid Serialization

✓ Chosen

Redis Lua Script

Single-threaded atomicity. 0.3-0.8ms per bid. No locks, no retries. 20-50x faster than DB. Cost: no durability — mitigated by sync replication + Kafka + PG snapshots.

✗ Alternative

PostgreSQL Row Locks

Full ACID. 5-20ms per lock (disk fsync). At 50 bids/sec on one auction, last bid waits 500ms+. Lock held during app crash for 30s. Correct but too slow.

Auction Closure Strategy

✓ Chosen

Hard Close + Gateway Timestamp Grace

Auction stops accepting bids at end_time. Bids that entered the Gateway before end_time are accepted within a 2-second window. Clean, fair, legally defensible.

✗ Alternative

Pure Hard Close (No Grace)

Simpler. eBay's approach. But penalizes bidders with higher network latency. A bidder in Mumbai is disadvantaged vs. one near the data center.

Async Event Bus

✓ Chosen

Kafka for All Async Processing

Decoupled consumers. Event replay on failure. Ordered by auction_id partition. Adding new consumers (analytics, ML) requires zero changes to Bid Service.

✗ Alternative

Direct HTTP Service-to-Service

Lower latency (2-5ms vs 5-15ms). Simpler topology. But tight coupling: every new consumer requires Bid Service changes. No replay. No buffering during outages.

Read/Write Consistency Split

✓ Chosen

CQRS — Strong Writes, Eventual Reads

Writes serialized through Redis master. Reads from replicas, caches, Elasticsearch. 1-2 second staleness on reads. Scales reads independently to 1000x write volume.

✗ Alternative

All Reads from Authoritative Source

Perfectly consistent reads. But 50K reads/sec per popular auction would overwhelm the Redis master. Can't scale reads without scaling the serialization point.

08

What Can Go Wrong

Redis Master Crash During Hot Auction

All in-memory state for ~500K auctions on that shard is lost. Mitigation: Sentinel failover to sync replica in 2-5s. Sub-50ms data loss window. In-flight bids get 503, clients retry with same idempotency key. Kafka event log enables state reconstruction.

Ghost Bid — Server Crash After Lua, Before Kafka Publish

User told "bid accepted" but event never reaches PostgreSQL or notifications. Mitigation: Bid receipt written inside Lua script (same Redis transaction). Retry checks receipt before re-processing. Reconciliation job detects Redis/PG divergence every 5 minutes.

Scheduler Leader Crash at Peak Closure Time

Auctions stop closing. 100 auctions/sec should be ending. Mitigation: Standby acquires leadership via Redis lease in 2-4s. Backlogged auctions mass-closed in parallel by shard. Worst case: 12-second delay, during which late bids are accepted.

Clock Skew Causes Premature Closure

Redis clock drifts 5s ahead → auctions close early → legal liability. Mitigation: Clock skew guard in every Lua script (reject if caller and Redis TIME differ > 5s). NTP monitoring with 100ms drift alert threshold. Affected auctions can be reopened with time extension.

Shill Bidding Detected After Settlement

Buyer overpaid $4K due to artificial inflation. Payment captured, seller paid. Mitigation: Freeze seller account. Replay legitimate bids to compute fair price. Partial refund from seller's payout reserve. Real-time fraud detection prevents most cases pre-close.

WebSocket Tier Complete Failure

2M+ connections drop. No real-time updates. Mitigation: Clients fall back to HTTP polling every 3 seconds. Bid submission (REST) is unaffected. WebSocket reconnection with exponential backoff + jitter. STATE_SYNC message on reconnect catches up missed updates.

09

Interview Tips

💡
Start with the serialization problem, not the architecture diagram.
"The core challenge is that bids on the same auction must be serialized. Let me show why, and how that constraint shapes everything." This demonstrates you understand the fundamental challenge, not just the component list.
Derive numbers, don't memorize them.
"500K closures/day × 8 bids = 4M/day ÷ 86K = ~50/sec average. But the peak on a single auction is what drives my architecture — 50 bids/sec on one key." The derivation matters more than the number.
🎯
Proxy bidding is your differentiator.
Most candidates cover bidding + closure. Few discuss the Vickrey-like mathematical resolution, composite sort keys for tie-breaking, or information leakage prevention. This level of detail sets you apart.
💡
Call out the hot-key problem proactively.
Don't wait to be asked. "This creates a hot-key problem — I can't shard within one auction. Here's my mitigation: bid coalescing, hot auction isolation to dedicated shards, and update throttling for watchers."
Summarize the system's personality in one sentence.
"Strongly consistent on the write path via single-threaded Redis Lua, eventually consistent on the read path via replicas and pub/sub. This split is the fundamental decision — everything else follows."
🎯
Mention settlement as a separate bounded context.
"There's an entire state machine for settlement — payment holds, escrow, disputes, chargebacks. The most interesting challenge is chargebacks after seller payout — mitigated by reserve accounts and real-time fraud scoring."
11

Evolution

How this design grows from MVP to planet-scale.

1

MVP — Single Server (0-10K Auctions)

PostgreSQL handles everything. SELECT FOR UPDATE for bid serialization. Direct WebSocket pushes from the app server. Simple setInterval scheduler. No Kafka — synchronous writes and notifications. Works fine at ~0.3 bids/sec total.

2

Growing Pains — Introduce Redis + Kafka (10K-1M Auctions)

Redis Lua replaces PG row locks for bid processing. Kafka added for async event processing. Read replicas for browse traffic. Separate WebSocket tier with Redis Pub/Sub. Elasticsearch for search. Basic fraud detection. Single Redis sorted set scheduler.

3

Scale-Out — Shard + Isolate (1M-10M Auctions)

Redis Cluster with 10-50 shards. Hot auction detection and migration to dedicated shards. Three-layer scheduler (PG → Redis sorted set → timing wheel). Bid coalescing. Update throttling. ML fraud detection. Multi-region reads, single-region writes.

4

Global Scale — Multi-Region Active-Active Reads (10M-100M)

Auctions assigned to nearest region for write serialization. Global read replicas with <50ms latency. Per-region schedulers. CDN-backed event streaming for viral auctions. Region-specific payment processors. Cross-region fraud aggregation.

5

Platform Evolution — Beyond Auctions (100M+)

Multi-format: auctions, fixed-price, reverse auctions, Dutch auctions. AI pricing recommendations. Dynamic reserves. Automated listing via computer vision. Integration marketplace for third-party tools. The auction engine becomes one module in a commerce platform.

Next up