Feature Store

"User's last 7-day click count" is a feature. Your training pipeline computes it from click logs in Spark; your serving pipeline computes it from a Kafka stream. The two implementations drift — one bug here, one rounding there. Models trained on offline features mispredict on different online features. Training-serving skew kills more ML systems than bad models.

A feature store is the single source of truth for features. One implementation, two access paths: batch (training) and online (serving). Production-grade ML at every serious company runs on one — Uber Michelangelo, Airbnb Zipline, Instacart, Lyft, all have their own. Feast and Tecton are the open + commercial standards.

Every feature has two homes:

• Offline store — historical feature values for every entity at every point in time. Stored in a data warehouse (BigQuery, Snowflake) or lakehouse (Iceberg, Delta). Used to construct training data by joining labels with feature values as of the prediction time.
• Online store — latest feature values per entity, sub-millisecond lookup. Stored in a KV store (Redis, DynamoDB, Cassandra). Used at serving time when the model needs features for a fresh prediction.

Both populated by the same feature transformation code. Same logic, two destinations. That's the magic.

The killer feature (pun) of a feature store is point-in-time joins. To build training data without leaking the future:

1. Take the labels: e.g. (user_id, did_purchase, timestamp) rows.
2. For each row, join the feature values as they were at that timestamp. Not the latest value — the historical value at that exact moment.
3. Train on the result.

Without this, your training data has features computed from data that didn't exist yet at prediction time. Model looks great offline; flops in production. Feature stores enforce correctness via SQL window functions (LAST_VALUE … RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) or temporal joins.

Feature stores look simple in slides ("just two stores!"). They die in production for three reasons:

1. Schema sprawl. Teams add features ad-hoc; nobody owns the catalog. After a year, you have 5,000 features, 30% duplicates with subtly different names. Discoverability dies. Mitigation: feature registry with mandatory metadata (owner, refresh, lineage), feature-of-the-month review.

2. Online/offline drift. The "same code" promise breaks the moment someone hot-fixes the streaming job without backporting to batch. CI must replay the same input through both pipelines and assert numeric equivalence (within float epsilon). Tecton enforces this; rolling your own you'll skip it once and pay forever.

3. Freshness vs cost. 15-min batch features are cheap; 30-second streaming features cost 100× more compute. Most teams over-stream. Rule: only stream the features the model proves benefit from at sub-minute freshness.

Recommendation algorithm uses a feature store to share user/item features between training + serving. News feed ranker pulls real-time engagement features. Leaderboard recomputes scoring features in a streaming pipeline.