Model Serving — Online vs Batch

You trained a recommendation model. Now what — does it predict in real time when the user opens the app, or do you precompute every user's recs nightly and read from a cache? Same model, completely different infrastructure. Get the wrong answer and you either burn 10× the compute or fail to personalize at all.

Online serving is sub-100ms inference per request. Batch scoring is hourly/daily prediction over millions of users at once. Streaming scoring is the middle ground. Each fits a different product surface.

A request arrives. To serve a prediction, the system must:

1. Fetch features from the feature store — sub-5ms for ~100 features.
2. Encode + transform — categorical lookups, normalization, embedding fetches.
3. Run inference — model forward pass, ~5-50ms depending on model size.
4. Post-process — top-K filtering, business-rule overrides, deduping.
5. Return — total budget often 100-200ms.

The whole thing scales with traffic. 10k QPS = 10k inferences/sec. At that rate, every millisecond per inference matters. Hence why high-performance model servers (TensorFlow Serving, NVIDIA Triton, TorchServe) exist.

Online sounds better — fresher predictions. It's also 10-100× more expensive per prediction. Use batch when:

• Predictions are stable over hours. "Top 100 movies for this user" probably doesn't change in an hour. Compute nightly, serve from a KV store, save 99% of inference cost.
• The product surface tolerates staleness. Email digest, "for you" widget on the homepage. Fine if it was computed 8 hours ago.
• The candidate set is small + known ahead of time. Score every user × every item once; serve top-K from cache.

Use online when:

• Context matters per request (current session, query, location).
• The candidate set is huge and changes (search ranking over millions of products).
• Latency-to-prediction matters (autocomplete suggesting words as you type).

Modern recommendation + search systems split serving into two stages — and use different modes for each:

Stage 1 — Candidate generation. From millions of items, narrow to ~hundreds. Often batch precomputed — a daily job runs ANN search over embeddings to build "top 200 similar items per user." Cached.

Stage 2 — Ranking. The 200 candidates are scored by a richer model online, with full real-time context. Output: ordered top 10 to show.

This split is everywhere — YouTube recommendation, Twitter feed ranking, Amazon search. Stage 1 takes care of "find relevant stuff" cheaply (batch). Stage 2 takes care of "rank what's relevant given right now" expensively (online).

Net cost: full ML model runs on hundreds of items per request, not millions. Latency stays in the 100ms budget. Personalization stays high because stage 2 is fully online with current context.

Typeahead does pure online inference (every keystroke). Recommendation algorithm uses two-stage (batch candidates + online ranking). News feed scores top-200 candidates online for each load.