Secrets Management
Secrets in code → secrets in vault → no secrets (workload identity). Rotation is a process, not a feature — automate it or it won't happen.
Deep dive
Hierarchy of pain
- Plain text in repo (worst).
- Encrypted in repo (SOPS, sealed-secrets).
- Vault-backed at runtime (KV, HashiCorp Vault).
- Short-lived, dynamically issued credentials (Vault DB secrets engine).
- No secret at all — workload identity / mTLS.
Aim for level 4 or 5 for new systems.
Rotation
A secret you never rotate is a secret you don't know is leaked. Define rotation cadence per secret class (API keys: 90 days; signing keys: 30 days with overlap; passwords: as needed). Test rotation in non-prod — finding out it breaks during a prod rotation is bad.
Real-world example
From productionA monorepo migration grep'd for AWS keys and found 17 in archived branches. Cleaning history was politically painful; all keys were rotated. New policy: pre-commit hook + Trufflehog in CI; secrets only ever live in Vault. Two years later: zero incidents from leaked credentials.
Interview questions
1 senior-levelQ1What's the best way to give a Kubernetes pod a database password?▾
Don't. Use workload identity to authenticate to a cloud DB, or have Vault issue short-lived per-pod credentials. If you must store a static password, use a secret manager (KV, Vault) accessed at runtime — not a Kubernetes Secret object alone, which is base64 not encrypted by default.
Common mistakes
Treating Kubernetes Secrets as encryption (they're base64).
Manual rotation that nobody owns.
Sharing one DB user across all services.
Trade-offs
Dynamic credentials add Vault as a hard dependency for app startup — plan for Vault HA.