Your recommendation service is down. Does your homepage return a 500, or does it just skip the "Recommended for you" section and still render? The first is a hard failure; the second is graceful degradation. Same failure, dramatically different user experience.
Graceful degradation is the discipline of designing so that partial-system failure produces partial-feature loss rather than total outage. Pairs tightly with circuit breakers.
When a downstream call fails, pick the best still-available response:
Each step down the ladder trades user experience for keeping the system responsive. The art is mapping each dependency to an appropriate fallback before it fails.
Feature flags for hot-disable. When a downstream starts misbehaving, flip a feature flag off instantly. The caller short-circuits to the fallback without deploying code. Stripe, LinkedIn, and Netflix all wire critical features this way.
Timeout-first, not retry-first. Every downstream call has a tight deadline (200ms, not 30s). If it's not back in time, skip it and use fallback. Don't let slow dependencies cascade into slow users.
Fallback is also tested. Most outages happen when the primary fails AND the fallback was never rehearsed. Chaos engineering (force-fail dependency X; verify fallback engages correctly) is how you keep fallbacks real.
Degrade cheap, not expensive. Serving trending-for-all to a million users is cheap; recomputing personalized recs synchronously is not. If the recs service is down, the fallback must be a pre-warmed global ranking, not a synchronous emergency calculation.
| Component | Normal | Fallback |
|---|---|---|
| Product title & price | DB lookup | Hard requirement — fail page if unavailable |
| Customer reviews | Reviews service | Cached; if > 24h old show anyway |
| "Frequently bought together" | Recs service | Hide the widget |
| Inventory / shipping ETA | Inventory service | "Usually ships in 1-2 days" static |
| User's order history link | Account service | Show link anyway; user clicks → handle failure there |
Only title + price is a hard dependency. Everything else has a fallback. Why Amazon rarely shows a blank product page even during outages.
For top-of-funnel pages (homepage, category browse), Amazon famously generates a static HTML fallback every few minutes. If the dynamic tier is overwhelmed, CDN/edge serves the static version — no personalization, but the site stays up. Users see something instead of 503.
Generalized: every dynamic page should have a stale-OK story.
Identify which pages must be fresh vs can survive staleness. Usually the answer is: most pages can. The hard real-time requirement lives only in transactional flows.
"For every synchronous call to another service, answer: if this fails, what do we show the user?" If the answer is "a 500 page," you haven't designed the degradation. If the answer is "we hide that widget and log the incident," you have.
If recs down → show trending. If artwork down → show default poster. If titles down → show 503 (rare; title metadata is replicated heavily).
Top-N product and category pages snapshot every few minutes to S3. CDN serves them when origin is overloaded.
Each feature has a kill switch. When a service misbehaves, an on-call engineer flips the flag and the feature disables cluster-wide in seconds.
If spellcheck down → skip "did you mean." If personalization down → show generic ranking. If core index down → 500 (no graceful option).
News feed falls back to "trending" if the ranker is slow. YouTube/Netflix falls back to default posters if metadata is stale. E-commerce hides optional widgets when their services degrade. Notification system uses channel-level fallbacks (SMS fails → email; email fails → in-app).
