Metrics & Monitoring

Two primary data paths: ingest path (agents push metrics through a gateway into Kafka, then TSDB writers compress and store in time-bucketed blocks) and query path (user queries fan out to TSDB readers across time shards, merge results). A separate alert evaluator runs on a cron loop, executing queries and firing notifications on threshold breach.

Ingest path detail: Each agent batches 100-500 samples and POSTs to the ingest gateway. The gateway validates the payload (metric name format, tag count limits, cardinality checks), authenticates via API key, and publishes to Kafka. Kafka is partitioned by hash(metric_name + sorted_tags) so all samples for the same series land on the same partition — this gives TSDB writers write locality. Writers consume from Kafka, buffer in memory for the current time block (2 hours), and periodically flush compressed chunks to disk.

Query path detail: The query gateway parses the expression, resolves label matchers against the metadata index to find matching series IDs, determines which time blocks overlap the requested time range, and fans out read requests to TSDB readers that own those blocks. Each reader decompresses the relevant chunks, computes partial aggregates, and streams results back. The gateway merges partial results and returns the final answer. For very wide time ranges, the gateway automatically routes to downsampled data.

Alert path detail: The alert evaluator is a stateful cron service that reads rules from a Postgres-backed rules DB. Each evaluation cycle, it partitions rules across evaluator instances (sharded by rule ID hash for horizontal scaling). Each instance runs its assigned rules as TSDB queries, tracks state transitions (ok/pending/firing/resolved), and publishes firing alerts to the notification service. The notification service handles deduplication, grouping (multiple related alerts into one page), and channel routing.

(a) TSDB storage model. Data is partitioned by time into blocks — typically 2-hour windows. Within each block, series are stored in columnar format with aggressive compression:

-- Gorilla encoding (Facebook, 2015):
-- Timestamps: delta-of-delta encoding
--   t0=1000, t1=1010, t2=1020, t3=1030
--   deltas: 10, 10, 10 → delta-of-delta: 0, 0, 0 → 2 bits each
-- Values: XOR encoding for IEEE 754 floats
--   v0=72.5, v1=72.6 → XOR = small diff → encode leading/trailing zeros
-- Result: ~1.37 bytes per data point (vs 16 bytes raw)

Compaction merges small blocks into larger ones periodically — for example, six 2-hour blocks merge into one 12-hour block with a single sorted index. This reduces the number of blocks the query engine must open for wide time ranges. Downsampling reduces resolution for older data: 1-min resolution for 7 days, 5-min for 30 days, 1-hour for 1 year. Downsampling pre-computes min/max/sum/count per interval so that queries on old data can still compute accurate aggregates. This keeps storage bounded while preserving long-term trends for capacity planning and SLA reporting.

(b) Cardinality explosion. Each unique combination of (metric_name, tag_key=tag_value) creates one time series. If someone adds user_id as a tag with 1M unique values — instant 1M series per metric. This kills ingestion rate, bloats the metadata index, and causes OOM on the TSDB.

// Bad: http.requests{user_id=U123456} → 1M series
// Good: http.requests{endpoint=/api/users, status=200} → ~500 series
// Solution: cardinality limit per metric (e.g., max 10K series)
//   + tag-value sampling for high-cardinality dimensions
//   + reject metrics that exceed cardinality budget at ingest gateway

(c) Alert evaluation loop. Every 15-60 seconds, the alert evaluator reads all active rules from the rules DB, executes each as a TSDB query, and compares the result against the threshold. For anomaly detection: compute rolling mean + standard deviation over a training window (e.g., same hour last 7 days), alert if current value exceeds mean + 3 sigma. The evaluator maintains state for each rule: pending (condition met but not yet for the required duration), firing (condition sustained past the for duration), and resolved (condition no longer met). State transitions trigger notifications.

// Alert evaluation pseudocode (runs every eval_interval):
// for each rule in active_rules:
//   result = tsdb.query(rule.expr, now - rule.window, now)
//   if result > rule.threshold:
//     if rule.state == "pending" && elapsed > rule.for_duration:
//       rule.state = "firing"
//       notify(rule.channels, "FIRING", result)
//     elif rule.state == "ok":
//       rule.state = "pending"; rule.pending_since = now
//   else:
//     if rule.state == "firing":
//       notify(rule.channels, "RESOLVED", result)
//     rule.state = "ok"

(d) Query fan-out and merge. A query like avg(cpu.usage{env=prod})[1h] requires reading data from multiple time blocks (e.g., if block size is 2 hours, a 1-hour query touches 1 block). But across 10K hosts, each block has 10K series entries. The query gateway identifies relevant series via the metadata index (inverted index: tag=value maps to series IDs), then fans out read requests to TSDB readers that own the relevant blocks. Each reader decompresses its chunk, computes a partial aggregate, and returns it. The gateway merges partial results into the final answer.

(e) Metadata index design. The metadata index is an inverted index: for each tag key-value pair, it stores the set of series IDs that contain that tag. Lookups are intersections of posting lists — identical to a search engine. For cpu.usage{host=web-01, region=us-east}, intersect the posting list for host=web-01 with region=us-east to find matching series IDs. This index lives in memory for fast lookup and is persisted to disk for durability.

// Inverted index structure:
// metric_name="cpu.usage"  → [series_1, series_2, ..., series_N]
// host="web-01"            → [series_1, series_47, series_902]
// region="us-east"         → [series_1, series_2, series_47, ...]
//
// Query: cpu.usage{host=web-01, region=us-east}
// → intersect([series_1,s_2,...], [s_1,s_47,s_902], [s_1,s_2,s_47,...])
// → [series_1]  ← only series matching ALL labels

• Push (Datadog) vs Pull (Prometheus). Push: agents POST metrics to a central gateway. Scales better for ephemeral containers (short-lived pods don't need to be discovered). Pull: monitoring server scrapes registered targets. Simpler for static fleets — no agent config needed, server controls scrape interval. Hybrid: Prometheus push-gateway for short-lived jobs.
• Pre-aggregation vs raw storage. Pre-agg (compute avg/p99 at ingest) saves storage and speeds queries, but loses the ability to re-aggregate differently later. Raw storage preserves flexibility — you can compute any aggregation at query time — but costs 10-100x more storage. Most systems store raw for recent data, pre-agg for older data.
• Purpose-built TSDB (Prometheus, VictoriaMetrics) vs general OLAP (ClickHouse). TSDB: optimized for time-range scans, Gorilla compression, built-in downsampling. OLAP: more flexible query language, handles higher cardinality, but less compression for regular time-series patterns. Datadog uses a custom TSDB; Uber Observability uses ClickHouse.
• Single global TSDB vs per-region shards. Global: simpler queries across regions. Per-region: lower ingest latency, data locality, but cross-region queries require fan-out and merge. Most production systems use per-region with a global query aggregator.
• Gorilla compression vs general-purpose (LZ4/Snappy). Gorilla achieves ~1.37 bytes/point for regular time series. LZ4 on raw floats gets ~4-5 bytes/point. Gorilla wins for uniform-interval numeric data; LZ4 wins for irregular or string data.
• Alert on metrics vs alert on logs. Metric alerts are cheap (evaluate one number against a threshold). Log-based alerts require full-text search per evaluation — 100x more expensive. Use metrics for known failure modes (CPU, latency, error rate); logs for unknown/exploratory debugging.
• In-memory hot tier vs disk-only. Keep the most recent block (current 2-hour window) in memory for fast writes and low-latency queries. Older blocks on SSD. Oldest blocks on object storage (S3). Tiered storage balances cost with query performance.
• PromQL vs SQL for metrics. PromQL is purpose-built for time-series: rate(), histogram_quantile(), label-based selection are first-class. SQL requires verbose CTEs and window functions for the same operations. But SQL is universally known — ClickHouse and TimescaleDB bet on SQL familiarity lowering the learning curve. Trade-off: expressiveness vs accessibility.
• Separate alert evaluator vs co-located with query engine. Separate process: can scale independently, failure doesn't affect user queries. Co-located: lower latency (no network hop for rule queries), simpler deployment. Most production systems use a separate evaluator for isolation.

1. Start with the data model. "A time series is identified by (metric_name + set of tag key-value pairs). Each series has a stream of (timestamp, float_value) tuples." This grounds everything.
2. Name Gorilla encoding explicitly. "Delta-of-delta for timestamps, XOR for float values — from Facebook's 2015 paper. Gets ~1.37 bytes/point vs 16 bytes raw." Shows depth beyond "we use a TSDB."
3. Cardinality is the interview differentiator. Most candidates describe ingest and query. Few mention cardinality explosion. Say: "The most dangerous production issue is unbounded tag cardinality — one bad deploy tagging with request_id creates millions of series and OOMs the cluster."
4. Separate ingest path from query path. Draw them as independent pipelines that share TSDB storage. This shows you understand read/write isolation — writers don't block readers.
5. Address "who monitors the monitoring?" Dead-man's switch: alert evaluator emits a heartbeat; a separate watchdog in a different failure domain checks it. This is the kind of operational nuance that impresses.
6. Mention push vs pull trade-off proactively. "Prometheus pulls, Datadog pushes. Push is better for ephemeral containers. Pull is simpler for static fleets. Modern systems support both via OpenTelemetry Collector."