Concept · Observability & Security

Secret Management

01

Why this matters

Your app needs a Postgres password, an AWS access key, a Stripe API key, a Twilio token, an OAuth client secret. Where do they live?

  • In code? Ends up in git history. Public repos leak credentials by the thousands every week.
  • In env vars? Better, but environment files often committed accidentally; ps -ef shows env on Linux; container logs can leak.
  • In a config file? Pasted around in Slack, copied to laptops, hard to rotate.

Secret management is the discipline of storing, distributing, and rotating credentials so that none of those bad outcomes happen. Plus dynamic secrets, audit logs, and short-lived tokens. Vault, AWS KMS+Secrets Manager, Doppler — the modern table stakes.

02

What a secret manager actually does

  • Encrypted-at-rest storage. Secrets stored encrypted using a master key (rooted in HSM or cloud KMS).
  • Authenticated retrieval. Apps fetch via API; access policies enforce who/what can read which secret.
  • Audit log. Every read is logged. Compliance + incident response.
  • Rotation. Programmatic password change — generate new secret, update database, distribute to apps. Rotate every 30-90 days routinely; immediately on suspected compromise.
  • Dynamic secrets. Generate per-request short-lived credentials (5-minute DB password). If leaked, they expire quickly. Vault's killer feature.
  • Workload identity. Apps prove who they are via cloud identity (IAM role, K8s ServiceAccount) — no chicken-and-egg problem of "what credential gets you the credential?"
03

The static-vs-dynamic axis

Static secrets

Stored, fetched, rotated periodically

Long-lived password / token. Stored in vault; fetched on app start. Rotated every 30-90 days. Most teams operate at this level. Stripe API keys, third-party API tokens, OAuth client secrets.

Dynamic secrets

Generated per-app per-request, short TTL

App asks vault for a DB password; vault creates one in Postgres with 5-min TTL; app uses it; vault auto-revokes on TTL. Compromise window is minutes, not months. Best practice when supported.

04

Deep dive — workload identity and the chicken-and-egg fix

"How does my app prove to Vault who it is?" Naive: give the app a Vault token in env var. But that token is itself a secret — same problem you started with.

Modern answer: workload identity. The infrastructure already knows who the app is:

  • EC2 / ECS / Lambda — IAM role attached to the compute resource. Apps call AWS metadata service to get a temporary token, signed by AWS, proving "I am role X."
  • Kubernetes — each pod has a ServiceAccount. Token mounted into pod filesystem, refreshed automatically. Vault validates the token with K8s API, then issues secrets.
  • SPIFFE / SPIRE — cross-platform standard. Each workload gets an X.509 cert tied to its identity; vault accepts the cert as auth.

Result: no app stores any long-lived credential. App identity flows from the platform → vault → DB credentials → DB. Rotation is automatic. Compromise of one app does not give the attacker any other app's secrets.

Interview answer

"Secrets live in HashiCorp Vault. Apps authenticate via Kubernetes ServiceAccount tokens (workload identity — no bootstrap secret). DB credentials are dynamic — Vault generates a short-lived Postgres user per request. Rotation is automatic and audited."

05

What goes wrong

  • Logging secrets. Apps log full request bodies; tokens end up in log aggregator. Strip credentials at log emission, not later.
  • Stack traces. Exceptions sometimes include connection strings. Sanitize.
  • Error messages. "Invalid auth: token=abc123..." in API responses. Never echo the token back.
  • Secrets in URLs. Tokens in query strings appear in browser history, server access logs, referer headers. Always use Authorization headers.
  • Forgotten rotation. Manual rotation never happens. Automate via vault rotation hooks; alert if any secret is older than policy.
06

Real-world

HashiCorp Vault

The reference

Self-hosted or HCP Cloud. Dynamic secrets, transit encryption, PKI, identity broker. Industry default for serious shops.

AWS Secrets Manager + KMS

AWS-native

Managed; integrated with IAM. Auto-rotates RDS / DocumentDB credentials natively. Simpler than Vault if you're all-AWS.

Doppler / 1Password Secrets

SaaS for app credentials

Polished UX for managing per-environment static secrets. Popular for smaller teams.

Sealed Secrets / SOPS

GitOps-friendly

Encrypt secrets, commit to git, decrypt at deploy time. Lower ops burden; less feature-rich than Vault.

07

Used in problems

Payment gateway uses Vault for all PSP credentials with mandatory rotation. E-commerce manages OAuth secrets per merchant integration. WhatsApp's media servers fetch encryption keys per request from a secret service.

Next up