Backup and disaster recovery basics: RPO, RTO, 3-2-1 and tested restores

Backups are only as good as the last restore

Almost every organisation has backups. Far fewer know how long a full recovery would take, how much data they would lose, or whether the backups would survive an attacker with administrator access. Those are the questions that matter on the day something goes wrong: a deleted database, a failed deployment, a cloud region outage or a ransomware attack.

Backup and disaster recovery (DR) are related but different. A backup is a copy of data you can restore from. Disaster recovery is the wider plan for bringing systems and services back, including infrastructure, configuration, people and communication. This guide covers both, starting with the two numbers that should drive every decision.

Start with recovery objectives

The US National Institute of Standards and Technology’s SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems, remains one of the clearest references on the subject. It defines three measures, set during a business impact analysis.

• Maximum Tolerable Downtime (MTD): the total time a business process can be disrupted before the impact is unacceptable to its owner.
• Recovery Time Objective (RTO): the maximum time a system can remain unavailable before there is an unacceptable impact on other systems, the business processes it supports and the MTD.
• Recovery Point Objective (RPO): the point in time, before the outage, to which data can be recovered from the most recent backup. In other words, how much data loss the business can tolerate.

NIST makes two points worth repeating. The RTO normally has to be shorter than the MTD, because after systems are restored there is usually extra work, such as re-entering or reprocessing data, before the business process is running again. And RPO is not part of MTD; it is a separate measure of data loss.

Setting them in practice

Recovery objectives are business decisions, not technical ones. For each important system, ask the people who depend on it:

• How long can we operate without this before customers, revenue or compliance are seriously affected?
• How much recent data could we afford to lose and re-create?
• Does the answer change at month-end, during a sale or at peak season?

The answers lead directly to technology choices. An RPO of 24 hours can be met with nightly backups. An RPO of minutes needs continuous replication or frequent log backups. An RTO of a day allows a rebuild from backups; an RTO of minutes needs a standby environment ready to take traffic.

The NIST guide sets out a seven-step contingency planning process: develop a policy, conduct the business impact analysis, identify preventive controls, create contingency strategies, write the plan, test and train, and keep the plan maintained. Small organisations can follow the same sequence in a lighter form.

The 3-2-1 rule

The US Cybersecurity and Infrastructure Security Agency (CISA) recommends the 3-2-1 rule as a trusted guideline:

• 3 copies of important files;
• on 2 different types of storage media, such as a hard drive and the cloud;
• with 1 copy stored off-site, away from your business location.

The idea is that no single failure (a disk, a building, a provider or a mistake) can take out every copy. CISA also advises choosing a backup solution that runs automatically and regularly, and protecting backups with physical security, encryption and offline copies.

Updating 3-2-1 for ransomware

Modern ransomware groups look for backups and try to destroy them before encrypting production systems. CISA’s #StopRansomware Guide therefore recommends maintaining offline, encrypted backups of critical data, noting that many ransomware variants attempt to find and delete or encrypt any backups they can reach. Practical ways to achieve that separation include:

• Immutable storage. Cloud object storage can enforce write-once-read-many retention. On AWS, for example, S3 Object Lock in compliance mode prevents any user, including the account’s root user, from overwriting or deleting a protected object version until its retention period ends. Object Lock requires versioning to be enabled on the bucket.
• Separate accounts and credentials. AWS’s DR guidance notes that copying backups to a different account helps protect against insider threats and account compromise. The credentials that run production should not be able to delete backups.
• Versioning. Object versioning keeps earlier versions when data is overwritten or deleted, which helps recover from human error.

CISA adds a caution that immutable storage does not meet the compliance criteria for certain regulations, so check retention requirements before you lock data.

Back up more than data

A database dump alone is not a recovery. To rebuild a service you also need:

• Infrastructure definitions. AWS’s disaster recovery whitepaper recommends deploying with infrastructure as code (IaC), because without it restoring a workload in another region can become complex enough to exceed your RTO.
• Application code and configuration, including machine images and container images.
• Secrets and keys, stored so that they can be recovered without depending on the system that failed.
• “Golden images”. CISA recommends maintaining and regularly updating golden images of critical systems, and keeping backups of IaC templates offline so resources can be redeployed quickly.
• Documentation and contacts, including runbooks, supplier support details and who has authority to declare a disaster.

SaaS data is still your responsibility

Moving to cloud services does not move responsibility for your data. Microsoft’s shared responsibility model states that across on-premises, IaaS, PaaS and SaaS, the customer always remains responsible for its data, configurations and identities. Check what retention and restore options each SaaS tool provides, and decide whether you need a separate backup of email, documents, CRM records and source code repositories.

Choosing a cloud DR strategy

AWS’s whitepaper describes four broad approaches, ordered from lowest cost and complexity to highest.

Two warnings from the same guidance are worth noting. First, continuous replication is not a substitute for backups. It may not protect against data corruption or malicious deletion as well as point-in-time backups do, so keep versioning or point-in-time recovery as well. Second, even an active/active design may need to fall back on backups after data corruption, which usually means a recovery point some time before the problem was discovered.

Use your RTO and RPO to choose. Many organisations use backup and restore for most systems and reserve pilot light or warm standby for the few that genuinely need it.

Test restores, not just backups

A backup that has never been restored is an assumption. NIST describes testing as a critical element of a viable contingency capability, and recommends testing recovery on an alternate platform from backup media, as close to a real operating environment as possible. CISA advises scheduled recovery tests to verify backup integrity, the ability to restore data both fully and partially, and the ability to roll back at least seven days.

AWS’s guidance goes further: it states that a backup strategy must include testing backups, and suggests automating periodic restores so you always have a recently restored, verified copy.

A practical testing programme:

1. Automated restore checks. Restore a sample of backups on a schedule, verify file counts and checksums, and start the database to confirm it is consistent.
2. Application-level tests. Restore a full environment, run the application against it and confirm key transactions work.
3. Timed recovery exercises. At least once a year, run a realistic scenario end to end and measure actual recovery time and data loss against your RTO and RPO.
4. Tabletop exercises. Walk through decisions, communication and escalation with the people involved, including who declares a disaster and who talks to customers.
5. Record and fix. Document what failed or took longer than expected, update the runbook and retest.

A short checklist

• Agree RTO and RPO for each important system with its business owner.
• Follow 3-2-1, with at least one copy offline or immutable and protected by separate credentials.
• Encrypt backups, and make sure the keys can be recovered independently.
• Back up infrastructure code, configuration, images and secrets, not just data.
• Confirm what each SaaS provider does and does not back up.
• Choose a DR strategy that matches your objectives, not the most expensive one available.
• Test restores on a schedule, and run a timed recovery exercise at least yearly.
• Keep the plan, contacts and runbooks current and accessible during an outage.

Sources

1. NIST SP 800-34 Rev. 1: Contingency Planning Guide for Federal Information Systems (publication page)
2. NIST SP 800-34 Rev. 1 (full text, PDF)
3. CISA: Back Up Business Data
4. CISA: #StopRansomware Guide
5. AWS: Disaster recovery options in the cloud (Disaster Recovery of Workloads on AWS whitepaper)
6. AWS: Locking objects with S3 Object Lock
7. Microsoft Learn: Shared responsibility in the cloud