Real-time Fraud Detection

Four services in the hot path: scoring service (orchestrator), feature store (online + offline), rule engine (fast short-circuit), model service (inference). A parallel streaming pipeline keeps online features fresh from the event bus. An offline training pipeline consumes labeled outcomes + features to retrain models weekly.

Hot-path in < 100 ms. The scoring service orchestrates:

1. Rule engine (1–5 ms). Short-circuit for obvious blocks — known-bad IP, sanctioned country, velocity thresholds exceeded. Rules are fast and deterministic; ML isn't needed to say "decline a card transaction from 10 countries in 10 minutes."
2. Feature fetch (10–30 ms). Parallel reads: online features (Redis), profile store (card history, user account age), graph store (device ↔ card ↔ IP clusters). All KVs keyed by entity_id.
3. Feature engineering (2–5 ms). Combine raw features into model inputs — ratios, deltas, categorical encoding. Done in-process in the scoring service.
4. Model inference (2–10 ms). GBDT (XGBoost/LightGBM) models dominate production fraud because they're fast, explainable, and perform well on tabular features. Neural nets occasionally used for specific signals (transaction-text embeddings, sequence models) but orchestrated alongside, not replacing.
5. Decision (1 ms). Compare score to thresholds (per-merchant). Return {score, decision, reason_codes}.
6. Log everything async (not on critical path): tx + features + score + decision to Kafka → offline store for training.

Feature freshness — the velocity problem. Fraudsters exploit time gaps. If a stolen card has been used 5 times in the last minute and we only see "lifetime card velocity" computed nightly, we miss it. Online features need sub-second updates.

Pattern: stream processor reads every transaction event; maintains sliding-window counts per entity (user, card, IP, device) at multiple granularities (1 min, 5 min, 1 hr). Written to Redis keyed by {entity_id}:{window}:{stat}. The scoring service reads these in parallel with the user/merchant profile.

Exact counts at entity granularity are fine (a user has few recent transactions). But aggregate counts across entities (e.g., "how many distinct cards used this IP in the last hour") benefit from Count-Min Sketches for bounded memory.

Graph features — finding rings. Single transactions look innocent; the ring doesn't. Graph features capture network-level signals: "how many other accounts share this device fingerprint?" "Has this IP been used by accounts that later charged back?" Stored in a graph DB or a materialized graph (adjacency lists) in Bigtable. Updated by a graph updater consuming the event stream.

Feedback loop. Three label sources:

1. Chargebacks (1–90 days later) — highest signal, lowest latency. Comes from payment network reports.
2. User reports — "I didn't make this transaction." Fast (hours), but noisy (users sometimes forget).
3. Analyst review — internal team labels borderline cases flagged by the model.

All labels flow into a join table (tx_id, features, label). A weekly training job pulls the last N days of labeled data, retrains GBDT, shadow-tests on recent traffic, then canary-deploys to a small % of tx. If metrics hold, promote to 100%.

The 1–90 day label delay is fundamental. You can't retrain on today's data today. Partial mitigation: heuristic labels (e.g., "3 chargebacks on this card in 30 days = probably fraud") extrapolated forward. Unsupervised anomaly detection supplements supervised on the most-recent window.

• GBDT vs deep learning. GBDT wins for tabular features: faster training, more explainable (per-feature contributions), runs in a few ms. Deep learning wins for unstructured inputs (text, sequence, images). Modern fraud systems use GBDT in the hot path; deep models run async for enrichment.
• Precision vs recall. Block more = catch more fraud + annoy more real customers. Every org picks a point on this curve based on business pain. Reviews (2FA) are the middle ground — expensive but not blocking.
• Rules + ML together. Pure ML is brittle for known-bad patterns (new fraud pattern, model hasn't seen it yet). Rules are maintainable by analysts. Hybrid is the default; don't propose "ML-only."
• Shared feature store vs per-model features. Shared store = consistent offline/online features across models (training-serving skew minimized). Per-model features = team autonomy, but drift risk. Large orgs go shared; small orgs can tolerate per-model.
• Synchronous block vs post-auth review. Synchronous needed for payment auth. Account security (logins, password resets) can tolerate asynchronous (score, flag for challenge on next action) — relaxes latency.

1. Rules + ML hybrid. Pure-ML is wrong. Mention rule engine short-circuit as a core design piece.
2. Online + offline feature store. Same features for training + serving. Training-serving skew is a classic production bug; show you know about it.
3. Label delay is fundamental. Acknowledge chargebacks take days-to-weeks. Don't propose "train on today's data, serve tomorrow" without caveats.
4. Graceful degrade. Payment auth can't hang. Fall back to rules + conservative threshold on any dependency failure. Payments > precision.
5. Graph features matter. Single-tx features miss rings. Shared-device-fingerprint clusters are a cheap and effective signal.
6. Explainability isn't nice-to-have. Compliance requires it. SHAP or tree-derived feature contributions are the standard.