Code Deployment Pipeline

Four tiers: Build tier (ephemeral containers compile + test), Artifact tier (S3 content-addressed store), Deploy tier (canary + progressive rollout), Observability tier (metrics collection + canary analysis). Git push triggers the pipeline end-to-end.

(a) Build Isolation. Each build runs in an ephemeral container spun from a base image per language (e.g., ci-golang:1.22, ci-java:21). The container has no outbound network access during build — hermetic builds. Dependencies are pre-fetched into a shared cache layer (content-addressed). Result: builds are fully reproducible. Same commit SHA = same artifact SHA, always. Container is destroyed after build; no state leaks between builds.

Why hermetic? Three reasons: (1) Reproducibility — build the same commit six months later and get the exact same binary. No "it worked on my machine." (2) Security — no network means a compromised dependency mirror can't inject malware at build time. (3) Cacheability — since builds are deterministic, unchanged modules produce the same output and can be skipped entirely (build-graph caching).

# Hermetic build: container spec
FROM ci-golang:1.22
COPY --from=dep-cache /deps /go/pkg/mod   # pre-fetched, content-addressed
COPY . /src
RUN --network=none go build -o /out/app   # no network access
RUN --network=none go test ./...           # tests also hermetic

(b) Artifact Store. S3-backed, content-addressed by the SHA-256 of the artifact. Key design: never overwrite. Once sha256:abc123 is written, it is immutable. Rollback is trivial: point the deploy config at the previous SHA. No rebuild required. Retention policy garbage-collects artifacts older than 90 days with no active deploy reference. Total store size ~50 TB across org.

Content-addressing gives you deduplication for free. If two services produce the same binary from the same source, they share one artifact in S3. The store is append-only — writes are idempotent (PUT with If-None-Match). Reads go through a CDN layer for geographic caching; a deploy in us-east pulls artifacts from the local edge, not cross-region. Metadata (build_id, commit, timestamp, SBOM) lives in Postgres; the blob itself lives in S3. Signed URLs ensure only authorized deploy agents can pull artifacts.

// Artifact store put — idempotent, content-addressed
PUT /artifacts/{sha256}
  If-None-Match: *          // only write if doesn't exist
  Content-Type: application/octet-stream
  Body: <docker image tar>

// Deploy config — just a pointer to a SHA
{ "service": "payments",
  "artifact_sha": "sha256:abc123",
  "replicas": 50,
  "canary_pct": 1 }

(c) Canary Analysis. When a new artifact is deployed, it receives 1% of production traffic. Over a 15-minute window, the canary analyzer collects p99 latency and error rate from both canary (new) and baseline (old) pods. It applies a Mann-Whitney U test (non-parametric, no normality assumption) to determine if the canary is statistically worse. If the test fails (p < 0.05 for regression), auto-rollback fires within 60 seconds. If the test passes, promote to 10%, repeat analysis, then 50%, then 100%.

Why Mann-Whitney instead of a simple threshold? A threshold like "error rate < 1%" is meaningless if your baseline already runs at 0.95%. Mann-Whitney compares the distributions of canary vs baseline samples. It detects relative regressions ("canary is statistically worse than baseline") regardless of absolute values. At 1% traffic over 15 minutes, you collect ~9K request samples — more than enough for statistical power. The test requires no normality assumption, which is important because latency distributions are heavily right-skewed.

# Canary analysis pseudocode
canary_latencies  = collect_p99(canary_pods, window=15min)
baseline_latencies = collect_p99(baseline_pods, window=15min)
stat, p_value = mann_whitney_u(canary_latencies, baseline_latencies)
if p_value < 0.05 and median(canary) > median(baseline):
    trigger_rollback()    # canary is worse; roll back within 60s
else:
    promote_to_next_stage()  # 1% → 10% → 50% → 100%

(d) Feature Flags. Decouple deploy from release. Code ships "dark" — the feature exists in the artifact but is gated behind a flag. The flag service evaluates rules (user ID % 100 < rollout_pct). Rollout: 1% → 5% → 50% → 100%, independent of the deploy pipeline. If the feature causes issues, flip the flag off — no redeploy needed. Flags are evaluated at runtime with ~1 ms overhead (local cache + SSE push for updates).

Architecture: a central flag store (Postgres) holds flag definitions + targeting rules. Each application instance caches the full flag set in memory (~10 KB). The flag service pushes updates via SSE — when an engineer changes a flag, all instances receive the update within 2 seconds. Evaluation is local (no network call per request). Stale flags are a real problem: after a feature is fully rolled out, the flag should be cleaned up. A "flag hygiene" job alerts on flags older than 90 days with 100% rollout — time to remove the conditional and delete the flag.

// Feature flag evaluation — local, no network
func isEnabled(flagKey string, userID int64) bool {
    flag := localCache.Get(flagKey)       // in-memory, ~0 ms
    if flag == nil { return false }       // unknown flag = off
    return userID % 100 < flag.RolloutPct // deterministic bucketing
}

• Canary vs Blue-Green. Canary: gradual traffic shift (1% → 100%), fine-grained analysis, but slower rollout. Blue-Green: instant swap between two full environments, fast rollback, but double infrastructure cost and no gradual validation.
• Hermetic builds vs cached network fetches. Hermetic (no network) = reproducible, secure, no supply-chain attack at build time. Cost: must pre-populate dependency cache. Cached fetches = faster cold start, but builds break if a dependency is yanked (left-pad incident).
• Statistical canary vs threshold-based. Threshold (e.g., "error rate < 1%") gives false positives when baseline is already at 0.95%. Mann-Whitney compares distributions directly — detects real regressions even when absolute values look fine.
• Feature flags vs deploy-gated releases. Feature flags add runtime complexity (flag evaluation on every request, stale flags accumulate). But they let you decouple risky feature rollouts from infrastructure deploys — a 10-line config change doesn't need canary analysis.
• Monorepo vs polyrepo CI. Monorepo: one pipeline triggers affected services (Google-style). Polyrepo: each repo owns its pipeline. Monorepo is simpler for cross-service changes but requires smart build-graph analysis to avoid rebuilding everything.
• Fast rollback vs fast forward. Rollback (re-deploy old artifact) is instant but means running old code with possible data-schema mismatches. Forward-fix (push a new commit with the fix) is slower but maintains forward progress. Best practice: rollback immediately to stop the bleeding, then forward-fix at normal pace.

1. Lead with immutability. "Artifacts are SHA-tagged and never overwritten. Rollback is just re-pointing to the previous SHA." This single idea simplifies everything.
2. Name the statistical test. "Mann-Whitney U for canary analysis" — shows you've thought beyond "just check if errors go up."
3. Distinguish deploy from release. "Deploy puts code on servers. Release exposes it to users. Feature flags let you do these independently." This nuance impresses.
4. Talk about blast radius. "Canary at 1% means worst case, 1% of users see a regression for 15 minutes before auto-rollback." Quantify the risk.
5. Mention hermetic builds. "No network during build. Dependencies from a pre-populated cache. Same commit = same artifact, always." Production-grade insight.