Cloud6 min read23 September 2026

Infrastructure as code, and the drift that quietly defeats it

Infrastructure as code delivers its promise only while the code describes reality. One console change during an incident, never reflected back, and the repository becomes documentation of what someone intended two quarters ago — at which point the next apply either fails or reverts a fix nobody remembers making.

Part ofCloud, Hosting & IT Infrastructure

The value of declaring infrastructure is not that provisioning gets faster. It is that the current state of a system is knowable by reading a file, reviewable before it changes, and reproducible somewhere else. All three properties depend on one condition: the file matches reality. Nothing enforces that condition by default.

Where drift actually comes from

Source	Why it happens	How long before it is noticed
Console change during an incident	Fastest path to restoring service	Until the next apply, often months
A permission granted to unblock someone	Opening a pull request felt disproportionate	Usually never
Provider-side defaults changing	The provider updated a default you never set	At the next plan, as a mystery diff
A resource created by another team	They had access and no reason to ask	When it appears as an unmanaged dependency
Autoscaling or managed-service mutation	The platform is supposed to change it	Not drift — needs excluding explicitly

Detect drift on a schedule, not at apply time

Discovering drift during a deployment is the worst possible moment: there is pressure to ship, the diff is unexpected, and the fastest resolution is usually to overwrite whatever was found. A scheduled check moves that discovery to a quiet Tuesday.

text

# Nightly, in CI, on a schedule rather than on push.
# -detailed-exitcode: 0 = no changes, 2 = changes present, 1 = error.
terraform plan -detailed-exitcode -refresh-only

# Exit code 2 means reality and code disagree. That is a finding to
# triage, not a failure to fix automatically:
#
#   - someone changed reality  -> decide whether to keep it, then
#                                 either codify it or revert it
#   - the provider changed      -> pin or set the value explicitly
#   - the platform mutates it   -> exclude it from management via
#                                 ignore_changes, with a comment
#                                 saying why
#
# Never auto-apply to resolve drift. The drift may be the only thing
# currently keeping production up.

State is the thing worth protecting

Most declarative tools keep a state file mapping declared resources to real ones. That file is not a cache — losing it means the tool no longer knows which resources it manages, and the next apply tries to create everything again alongside what already exists.

Remote backend, never local. A state file on a laptop is a single point of failure with no backup and no audit trail
Locking enabled, so two simultaneous applies cannot interleave and corrupt it
Versioned storage, because state corruption is recoverable only if a previous version exists
Encrypted at rest. State frequently contains secrets that were never meant to be stored — connection strings, generated passwords, keys
Separate state per environment. Shared state means a staging apply can reach production resources

The secrets point catches teams out. A generated database password or an access key appears in state in plaintext even when it never appears in the code, so state storage inherits the same protection requirements as any credential store.

Adopting it on infrastructure that already exists

Greenfield adoption is easy and rare. The usual situation is a running system built by hand, where a rewrite is unacceptable and a big-bang migration is how outages happen.

Start with the safest layer. Networking, DNS, storage buckets and IAM are mostly declarative already and rarely stateful.
Import rather than recreate. Every tool has an import path; using it means the resource is managed without being touched.
Verify the plan is empty after each import. A non-empty plan means your code does not match reality yet, and applying it would modify a live resource.
Leave databases until last, and import them rather than ever letting a tool create a replacement.
Add a policy that new resources are created in code only. Otherwise the hand-built surface grows faster than you import it.

Structuring it so it stays reviewable

One directory per environment, with the differences visible rather than hidden behind conditionals
Modules for genuine repetition only. A module wrapping one resource used once adds indirection and removes clarity
No conditional logic deciding what exists. Infrastructure that appears based on a variable is infrastructure nobody can read the shape of
Plan output in the pull request, because the diff in the code is not the diff in the world
Pinned provider and module versions, so an unrelated release does not change a plan you already reviewed

The second point is where most codebases go wrong. Premature modularisation produces a repository where answering "what does production actually look like" requires tracing four layers of variable indirection — which defeats the readability that was the main reason for declaring it.

What working infrastructure code looks like

A scheduled drift check that has reported no drift for weeks, because drift is reconciled when it appears.
Every plan reviewed by a second person before apply, with the plan output attached to the change.
A new environment provisionable from the repository, and that claim tested at least once rather than assumed.
Console access that is read-only for most people most of the time, with elevation available for incidents.
An incident checklist whose final item is reconciling any console change back into code.

Item three is the honest test of whether the repository describes the system. A team that has never built a second environment from its own code does not know whether it could — and the gap between the repository and reality is usually discovered at the least convenient moment.

Questions

How do we stop configuration drift?

Detect it on a schedule rather than at apply time. A nightly refresh-only plan surfaces disagreement between code and reality on a quiet day instead of mid-deployment, when the pressure is to overwrite whatever was found. Then add one step to your incident close-out: reconcile any console change back into code. The console fix during an outage is legitimate; not returning afterwards is what creates drift.

Should drift be corrected automatically?

No. An automated apply that reverts an undocumented change can take down the system that change was fixing. Drift detection should raise a ticket or a notification and let a person decide whether reality or the code is correct — sometimes the drift is the only thing keeping production up, and sometimes it is an unauthorised change that should be reverted. Only a human can tell which.

How do we adopt infrastructure as code on a system built by hand?

Import rather than recreate, starting with the safest layer — networking, DNS, storage, IAM — and verify that the plan is empty after each import, since a non-empty plan means applying would modify a live resource. Leave databases until last and never let the tool create a replacement for one. Add a policy that new resources are created only in code, or the hand-built surface grows faster than you import it.

Why does the state file need so much protection?

Because it is the only record of which real resources the tool manages. Lose it and the next apply tries to create everything again alongside what exists. It also routinely contains secrets that never appear in the code — generated passwords, access keys, connection strings — so it needs a remote backend with locking, versioning and encryption at rest, and separate state per environment.