Load Balancer

A single server can serve maybe a few thousand requests per second before CPU, memory, or connection limits pin it. The moment your traffic crosses that ceiling, you have two options: buy a much bigger server (vertical scaling — expensive, has a hard ceiling, single point of failure) or put many cheap servers behind something that spreads traffic across them. That "something" is a load balancer.

Without a load balancer, horizontal scaling is impossible. Clients would need to know every backend server by IP, pick one, and retry when it's down. The load balancer hides the fleet behind one stable address and deals with failure, capacity, and routing on the client's behalf.

Picture a grocery store on Sunday afternoon. Twelve checkout lanes are open. You don't walk in and pick lane 7 because you memorised it — you look for the shortest queue, or a greeter directs you to the next free cashier. If a cash register breaks, the greeter stops sending you there. If the store gets busier, they open more lanes.

The greeter is a load balancer. The cashiers are your backend servers. The shoppers are your requests. The whole point: the shoppers don't need to know which cashiers exist — they just enter the store and get served.

A request arrives at the load balancer's IP. The LB does four things:

1. Accepts the connection (TCP SYN/ACK) and optionally terminates TLS so it can read HTTP headers.
2. Picks a backend using a chosen algorithm (see §4) — round robin, least connections, IP hash, etc.
3. Opens or reuses a connection to that backend and forwards bytes. The client has no idea which backend answered.
4. Watches health continuously: every healthCheckInterval (often 2–5s), the LB pings each backend on a known path like /health. Three failed checks in a row → the backend is yanked out of rotation. The next time a check succeeds, it's put back.

There's a subtle detail here: health checks decide fleet membership, not the request router. The router only picks from backends the health checker has marked healthy. This decoupling is why a load balancer can remove a crashing server in 5 seconds without the router needing to know anything about crashes.

The single biggest decision is what layer you operate at. Everything else flows from it.

Once you've picked a layer, you pick an algorithm for distributing load across backends:

Load balancers buy you scale and availability, but they cost:

• Latency — an extra network hop (~0.1–1ms inside a region, worse across AZs).
• Cost — an AWS ALB at 1M requests/hour runs roughly $20/day per LB. Not free.
• A new failure mode — the LB itself. Mitigate with multiple LB instances, DNS round-robin between them, and anycast IPs.
• Stickiness headaches — if your backends hold session state in memory, the LB has to route the same user to the same server (cookie-based sticky sessions). This defeats the point of a fleet. Fix the stickiness at the backend layer — make services stateless and put session state in Redis.

You can skip a load balancer when:

• Traffic is low enough that one server handles it with room to spare and you don't need HA.
• The clients are inside your VPC and you can use client-side load balancing — the client itself has the backend list (via service discovery) and picks one. This is how gRPC load balancing and Netflix's Ribbon work. No dedicated LB in the path.
• DNS round-robin is enough (e.g., read-only static content, accept slow failover).

Imagine 1000 requests arriving at 100 backends. With random placement, the most-loaded backend ends up with about log N / log log N ≈ 5–6 requests when the average is 10. That 50–60% overload on the hot backend is the tail latency you see.

With round robin, load is perfectly balanced if every request costs the same. They don't. One slow query on server 3 makes round robin keep sending it more work.

With least connections, the LB tracks the exact load of every backend. That's O(N) state per decision, and requires constant updates — expensive at scale.

Power of Two Choices (Mitzenmacher, 1996): pick any two backends at random, send the request to the less-loaded of the two. With almost no state, the max load drops from log N / log log N to log log N / log 2 — for N=100, that's ~3.3, nearly half of random. The math is surprising and it's called the "power of two" exactly because two choices already captures most of the gain; going to three barely helps.

Real systems go one step further — P2C + EWMA (exponentially-weighted moving average of response time). Instead of "connection count," each backend tracks its recent latency. The LB picks two random backends and routes to the one with lower EWMA. This is what Envoy's LEAST_REQUEST with v2_least_request_lb_config.choice_count=2 actually does.

The sidebar shows every problem page in this portfolio where a load balancer sits in the architecture. Nearly every high-traffic system uses one — the exceptions are single-tenant admin tools and batch pipelines.