Deploys that can be undone, and the ones that cannot

Every deployment approach is a position on one trade-off: how much exposure a bad version gets before it is noticed, against how much infrastructure and discipline the approach demands. Choosing sensibly requires knowing which failure modes each one actually prevents.

The four approaches, compared honestly

Blue-green and canary solve different problems, which is why the choice between them is not about sophistication. Blue-green makes the reversal instantaneous but exposes everyone the moment you cut over. Canary limits who is exposed but means running two versions simultaneously, which your data layer has to tolerate.

Database migrations are what break rollback

Application code rolls back cleanly because it is stateless. Schema changes do not, because the old code cannot read the new shape and the data has already moved. This is the single reason most rollback plans do not work.

-- Cannot be rolled back. The old version breaks immediately on
-- deploy, and reversing the migration loses every value written
-- since it ran.
ALTER TABLE users RENAME COLUMN email TO email_address;

-- Expand and contract, over three deploys. Every intermediate
-- state is compatible with the version before and after it.

-- Deploy 1 (expand): add, do not remove. Old code ignores it.
ALTER TABLE users ADD COLUMN email_address text;
-- backfill, then keep both columns written by the application

-- Deploy 2 (migrate reads): new code reads the new column.
-- Rollback is still safe, because the old column is still current.

-- Deploy 3 (contract): only once nothing reads the old column,
-- and only after deploy 2 has been stable long enough that you
-- would not roll back past it.
ALTER TABLE users DROP COLUMN email;

• Additive changes are safe — a new nullable column, a new table, a new index
• Destructive changes need a separate later deploy — drops, renames, type narrowing, adding NOT NULL
• Every intermediate state must be readable by both versions, since during a rolling deploy both are running
• Deploy 3 is the one that gets forgotten, leaving a permanently half-migrated schema that the next person has to decipher

Separate deploying from releasing

Deploying ships code. Releasing exposes behaviour. Conflating them means every change to what users see requires a deployment, and every deployment risks changing what users see. A flag decouples them, which changes the risk profile of shipping entirely.

• Disable a broken feature in seconds, without a build, a deploy or a rollback
• Ship incomplete work safely, behind an off flag, instead of on a long-lived branch that diverges
• Enable for a subset first — internal users, then a small percentage — which is canary at the feature level rather than the deploy level
• Keep the flag evaluated server-side where the feature has security or data implications, since a client-side flag is a suggestion

What to verify before calling a deploy healthy

"The deploy succeeded" means the process exited zero. It says nothing about whether the application works, and the gap between those two facts is where most incidents begin.

1. 01A real health check, which exercises the dependencies the service needs — not an endpoint returning 200 because the process started.
2. 02Error rate compared against the pre-deploy baseline, not against an absolute threshold. A rise from 0.1% to 2% breaches no alert and is a serious regression.
3. 03Latency at the 95th percentile, per version where possible, because a change that doubles response time rarely triggers an error.
4. 04One critical user journey, executed end to end. A synthetic check that signs in and completes the primary action catches what no infrastructure metric reveals.
5. 05Saturation of the constrained resources, since a change in connection or memory behaviour manifests minutes after deploy rather than immediately.

Point two is where automated rollback earns its place. A deploy gate comparing error rate and latency against the previous version for a few minutes, with automatic reversal on regression, catches the large majority of bad deploys before anyone opens a dashboard.

Choosing, for a system of ordinary size

The last row is the one to take away. The deployment strategy determines who is exposed to a bad version; the migration discipline determines whether you can undo it at all. Teams routinely invest in the first and skip the second, then discover during an incident that the sophisticated rollout has nothing to roll back to.