systemd Update

Datadog's VMs had Ubuntu unattended-upgrades enabled. An Ubuntu security patch to systemd-networkd shipped; VMs auto-updated; the update restarted networking on each VM in a way that broke CNI config for the running K8s pods. Every region affected simultaneously because unattended-upgrades runs on a cron schedule that doesn't vary by region. Datadog, whose entire business is "observability you can trust," went dark — customers couldn't see their own systems' metrics during the outage.

• Pre-event — Datadog VMs run Ubuntu with unattended-upgrades enabled — standard security best practice.
• ~06:00 UTC Mar 8 — Ubuntu pushes systemd-networkd security patch.
• ~06:00 UTC — Datadog VMs across 5 regions begin applying the patch as cron runs unattended-upgrades.
• 06:00–06:30 — systemd-networkd restart on each VM removes K8s CNI config. Pods lose networking. Datadog internal services cascade.
• 06:30–08:00 — Datadog engineers realize scale of impact. Disable unattended-upgrades everywhere. Build + ship a fix that re-programs CNI.
• 08:00–24:00 — Gradual region-by-region recovery. Service fully recovered ~24 hours later.

The specific bug: Ubuntu's systemd patch triggered a "reload" of networking that doesn't actually cooperate with Kubernetes CNI plugins. CNI sets up virtual interfaces + iptables rules at pod-creation time. systemd-networkd reload cleared those rules. Pods now had interfaces but no routing.

The deeper cause: unattended-upgrades was running across ALL VMs simultaneously. Multi-region redundancy didn't help because the update landed everywhere at the same hour. "Multi-region" was a geography strategy, not a time-schedule strategy.

24 hours of Datadog unavailability, globally. Customers operating critical infrastructure couldn't see their own metrics, alerts, logs, or APM traces. Estimated ~18,000 paid customers impacted. Service credits + customer trust hits. The irony — a monitoring company blind during its own outage — led to significant internal process changes.

1. "Multi-region" must include "multi-time." Regional redundancy is useless against a synchronized global cause. Stagger auto-updates by region.
2. Auto-updates on production are a double-edged sword. Security team loves them. Infra team hates them. The real answer: auto-updates with staged rollouts + the ability to pause if something breaks.
3. Monitoring systems need special care. A monitoring platform is mission-critical to customers by definition. Its own reliability bar has to exceed the bar of the systems it monitors.
4. Treat the OS as a dependency. OS + distribution + kernel + init system is part of your stack. Upgrades are code changes. Canary + monitor them.

• Deployment strategies — auto-upgrades need canaries too.
• Multi-region — redundancy across space doesn't save you from a synchronized cause.
• Kubernetes — CNI + systemd coupling.
• Monitoring — the meta-problem of monitoring your monitors.
• Chaos engineering — "what if all regions update at once?" is a testable scenario.