Secrets management, and what to do the morning a key leaks

Secrets management is usually framed as a storage problem. It is really a lifecycle problem: a credential you cannot rotate quickly is a credential you are permanently exposed by, regardless of how well it is stored.

Where secrets should live, best to worst

Most teams can move one tier up the table in an afternoon, and that is usually the right amount of ambition. Going from plaintext to platform environment variables is a genuine improvement; waiting for a full vault deployment while the key sits in git is not.

Rotation is the only property that matters

A secret's real security property is the time between deciding to revoke it and it being revoked everywhere. If that number is days, you do not have secrets management — you have secrets storage.

Rotation without downtime requires that two credentials can be valid at once. Design for that from the start; retrofitting it during an incident is how outages get added to breaches.

Rotation, in the only order that avoids downtime:

1. Issue a NEW credential. Both old and new are now valid.
2. Update every consumer to the new one. Deploy, confirm, wait.
3. Verify the old credential has no remaining use — check provider
   access logs, not your own inventory. Inventories are wrong.
4. Revoke the old credential.
5. Confirm nothing broke. If something did, it was using the old one
   and step 3 missed it — that is the finding worth keeping.

The reverse order (revoke, then update) is the classic self-inflicted
outage: everything breaks between steps, and under incident pressure
someone re-enables the compromised key to restore service.

Detection: assume it will leak

Prevention is necessary and insufficient. Keys leave through commits, logs, error reports, screenshots, support tickets and pasted terminal output. The goal is to find out from your own tooling rather than from a third party.

1. 01Pre-commit scanning — a local hook that refuses to commit a recognisable credential. Cheapest possible intervention, catches the majority of incidents before they exist.
2. 02CI scanning of the full history — because the hook was added after the mistake, and a leak in an old commit is still a live leak.
3. 03Provider-side alerting — most cloud providers can alert on anomalous credential use. This is the control that catches the key you did not know existed.
4. 04Scan your own logs — grep the log pipeline for credential patterns on a schedule. Secrets in logs are extremely common and almost never noticed, because nobody reads logs looking for them.

Scope every credential down

The damage a leaked credential does is bounded by what it can do. Most leaked keys turn out to have far broader permissions than the workload ever used, because the permissions were set once during a debugging session and never narrowed.

• One credential per workload, never one shared across services — a shared key cannot be rotated without coordinating every consumer
• Read-only wherever the workload only reads; the majority of integrations only read
• Scoped to the specific resources needed, not the whole account or project
• Network-restricted where the provider supports it, so a key exfiltrated elsewhere is useless
• Short-lived by default, with expiry as a backstop against a revocation nobody ran

A useful audit: for each long-lived credential, ask what it is permitted to do and what it has actually done in the last ninety days. Provider access logs answer the second question. The gap between the two answers is free risk reduction.

Local development is where discipline leaks

Production secrets are usually governed. Development is where a real key gets copied into a file, shared in a chat message to onboard a colleague, and carried on a laptop for two years.

# .env.example — committed. Names and shapes only, never values.
DATABASE_URL="postgres://user:password@localhost:5432/appdev"
CONTACT_WEBHOOK_URL=""        # optional; form logs instead if unset
SMTP_PASSWORD=""              # obtain from the secret manager

# .env.local — gitignored, never committed, never shared in chat.
# Each developer fetches their own values from the secret manager.

# .gitignore
.env*
!.env.example

• Developers get their own credentials against non-production data, never a copy of a production key
• The committed example file documents names and shapes so onboarding needs no credential sharing
• Negated gitignore pattern, so the example is committed and every real variant is not
• A credential that arrives over chat is compromised the moment it is sent — the chat history retains it indefinitely

The morning a key leaks

1. 01Rotate first. Investigate second. The instinct is to assess the exposure before acting; every minute spent assessing is a minute the credential is still valid.
2. 02Pull the provider's access logs for that credential and look for use from unfamiliar addresses, at unusual hours, or against resources the workload never touches.
3. 03Determine what the credential could reach, and whether any of it constitutes personal data — that question decides whether a notification obligation exists, and the clock is short.
4. 04Clean the history, add the detection control that would have caught it, and record the actual rotation time.
5. 05Write the incident note while the detail is fresh. The finding is almost never "someone was careless"; it is that the credential was over-scoped, long-lived, and nothing was watching.

Judged properly, a leaked credential is a test of the lifecycle rather than of the individual. A team that rotates in four minutes has a bad afternoon. A team that cannot rotate has an incident that lasts as long as the key does.