Cloud6 min read23 September 2026

Backups you have actually restored, and the ones you only hope exist

Every organisation has backups. A much smaller number have restored one. The distance between those two positions is usually discovered on the worst day of the year, when someone finds that the nightly job has been failing silently since a credential rotation in March.

Part ofCloud, Hosting & IT Infrastructure

A backup is a file. A recovery capability is a tested, timed procedure that a named person can execute under pressure with the documentation available. Most organisations have invested in the first and assumed the second, and the assumption is not usually correct.

Two numbers, chosen deliberately

Every recovery design is determined by two figures, and they should be decided by whoever owns the business consequence rather than inferred by whoever configures the storage.

	Question	Determines
RTO — recovery time objective	How long can this be unavailable?	Whether you need standby infrastructure or can rebuild
RPO — recovery point objective	How much recent data can be lost?	Backup frequency, and whether continuous replication is needed

The two are independent, and conflating them produces the wrong architecture. A system that can be down for a day but must lose no transactions needs continuous capture and no standby capacity. One that must be back in five minutes but can lose an hour of data needs the opposite. Asking both questions separately is most of the design work.

The failure modes a backup does not cover

Backups are designed against hardware failure and accidental deletion. Several of the most likely failures are not in that category, and each needs a different control.

Failure	Does a nightly backup help?
Storage or instance failure	Yes — this is the case it was built for
Accidental deletion of records	Yes, if noticed within the retention window
Slow corruption over weeks	Only if retention reaches back past the corruption
Ransomware encrypting the estate	Only if a copy is immutable or offline
Credential compromise with delete access	Only if backups are outside the compromised account
Account closure or regional loss	Only if a copy exists outside that account and region
Provider-side deletion of a managed service	Only if you export, rather than relying on their snapshots

The slow-corruption row is worth dwelling on, because it defeats short retention entirely. If a bug has been quietly writing bad data for six weeks and your retention is thirty days, every available backup contains the corruption. Retention has to span the time it plausibly takes to notice a subtle problem, not the time it takes to notice an outage.

The distribution rule, stated plainly

Three copies of anything that matters — the live data and two others.
Two different storage media or services, so one product's failure mode does not take both.
One copy off-site, meaning a different region and ideally a different account.
One copy immutable or offline, which is the modern addition and the one that addresses ransomware.
Zero unverified restores, which is the only clause with a failure condition you can test today.

The last clause is the point of the whole list. The first four describe an arrangement of files; only the fifth establishes that the arrangement works.

The restore drill

Run this against a real backup, into an isolated environment, with the documentation and nothing else. Do it before you need it and time every step.

text

Restore drill — quarterly, timed, written down

 1. Pick a backup nobody prepared for this. Yesterday's, at random.
 2. Follow only the written runbook. If a step is missing, note it
    and stop the clock — that gap is the finding.
 3. Restore into an isolated environment. Never over production,
    and never into anything sharing its credentials.
 4. Verify the data, not the process: row counts against expectation,
    the most recent records present, referential integrity intact,
    one real user journey completed against the restored copy.
 5. Record the actual elapsed time, and compare it to the stated RTO.
 6. Record what was missing: an undocumented step, a credential
    nobody had, a dependency nobody listed, a script that assumed
    something no longer true.

The drill's output is not "the restore worked". It is the list from
step 6, which is the difference between a policy and a capability.

Monitor the backup, not just the schedule

Alert on absence, not only on failure. A job that stops running entirely emits no error — the alert must fire when a backup has not appeared, not when one reports a problem
Check the size. A backup at a tenth of yesterday's size completed successfully and captured almost nothing
Verify restorability automatically where the platform allows it, rather than only during the quarterly drill
Alert on retention drift, so an expiry policy change does not silently shorten your recovery window
Include backup health in the on-call dashboard, because a backup failure discovered three months later is an unrecoverable window

The first item is the one that catches the classic failure. A credential rotation breaks the backup job, the job stops running rather than failing, and the monitoring — which only watches for failures — reports nothing wrong for months.

Write the runbook for the worst case

Assume the person recovering the system is not the person who built it, is doing it at 4am, and cannot reach anyone. That assumption changes what the document needs to contain.

Where the backups are, in full, including account and region — not "in the usual place"
Which credentials are needed and where they are held, with a path that does not depend on the system being down
The exact commands, copy-pasteable, with the dependency order stated
How to verify success, specifically, rather than "check it worked"
Who to notify, and what any contractual or regulatory notification clock requires
Stored somewhere that survives the outage — a runbook hosted only on the failed system is not available

That last point sounds like a joke and is a recurring real failure. Recovery documentation in a wiki running on the infrastructure being recovered is documentation you cannot read at the only moment it matters.

Questions

What is the difference between RTO and RPO?

RTO is how long the system can be unavailable; RPO is how much recent data can be lost. They are independent, and conflating them produces the wrong architecture. A system that can be down a day but must lose no transactions needs continuous capture and no standby capacity; one that must return in five minutes but can lose an hour needs the opposite. Decide both per system, with whoever owns the business consequence.

Do backups protect against ransomware?

Only if at least one copy is immutable or held outside the compromised account. An attacker with sufficient permissions deletes the backups first, because that is what makes the extortion work. A write-once copy with a fixed retention period, or a copy in a separate account with separate credentials, is what turns one blast radius into two.

How often should we test a restore?

Quarterly, timed, from a backup nobody prepared, into an isolated environment, following only the written runbook. Verify the data rather than the exit code — row counts, the newest records present, referential integrity, one real user journey against the restored copy. The output that matters is the list of things the runbook was missing, not the fact that it eventually worked.

Why do backup jobs fail silently?

Because monitoring usually watches for failure reports, and a job that stops running entirely — after a credential rotation, a permission change or a renamed resource — emits no error at all. Alert on the absence of a recent backup rather than on error events, and check the size too: a backup at a tenth of yesterday's completed successfully and captured almost nothing.