Testing backup recovery in practice

A green backup job status proves only that the program wrote some data and reported no error. It does not prove that the organization can return a working service to the required point in time, meet the allowed deadline, or find the passwords, keys, and instructions on the day of an incident.

A reliable test consists of two distinct jobs: automated storage integrity checks and regular trial recovery in an isolated environment. It needs a third layer that teams often forget: application acceptance of the restored service. If the accounting database starts but the latest finalized documents are missing, the technical recovery passed while the business recovery failed.

A successful backup does not mean a successful recovery

Test the whole chain that returns the service, not the fact that a backup was created. That chain includes the medium or object storage, backup catalog, encryption, credentials, configuration, recovery software version, application dependencies, and the person who accepts the result. One unavailable element breaks the chain.

It helps to separate three claims that reports often mix together. Integrity means that the stored blocks can be read and match their checksums. Recoverability means that the selected point can produce files, a virtual machine, or a database. Service readiness means that the application starts, users can sign in, data is consistent, and external dependencies connect in a controlled order.

The PostgreSQL documentation for pg_verifybackup states the limitation plainly: the utility compares files with the manifest and checks the required WAL records, but it cannot perform every check that a running server will make. The developers explicitly recommend test restores and checks that the data is correct. That is a sound rule for any backup product: a built-in Verify button is necessary, but it does not certify service recovery.

NIST SP 800-53 makes the same separation in control CP-9. It distinguishes testing media reliability and information integrity from restoring a sample of system functions. This distinction helps people beyond audit teams. It prevents someone from closing the task with a screenshot of green checkmarks when nobody has ever started a restored instance.

The test scope must match the recovery scope

Before setting schedules or commands, create a short recovery profile for every significant service. The statement "we back up the database server" is too narrow. An incident may also require DNS, a service account, a certificate, application configuration, a message queue, a license, and instructions for changing an address.

The profile only needs to record the service owner, the data and dependency scope, the allowed data loss or RPO, the allowed downtime or RTO, the method for selecting a recovery point, and the acceptance criteria. RPO answers how old the recovered information may be. RTO sets the deadline for returning the function, not merely the time spent copying files. If the environment started in two hours but access approval took three more, the actual RTO was five hours.

Test these boundaries against a specific event. For example, a request system requirement might read: "after losing the primary site, restore confirmed requests no more than 30 minutes old, open read access to operators within four hours, then allow writes after reconciling the queue." That statement immediately shows which transaction logs are needed, who accepts the data, and when the service can count as recovered.

Do not choose only a convenient file share for an exercise. The backup catalog should map each critical process to recoverable components. Once a quarter, work from the business function to the backup rather than the other way around: name the function, find all its dependencies, and prove that each one appears in the plan. This exposes forgotten encryption keys, load balancer configurations, and small databases that have lived "temporarily" for years.

Record the exercise assumptions separately. If the operator uses the live management server, a domain account, and the internal knowledge base, the test quietly assumes all three remain available. For a serious scenario, run at least some exercises from the emergency kit: a clean workstation, an offline procedure, separate credentials, and a copy of the required software.

Automation must read data, not just the catalog

A daily check should find missed jobs, a last recovery point that is too old, an unusually small volume, catalog errors, and expired retention. This is the fast control. It should end with a machine status and a notification to the owner, but it cannot replace reading the content.

Reading every large repository each day may consume the entire maintenance window. Split it into nonoverlapping portions and cover the full volume within a fixed cycle. In restic, for example, restic check checks metadata, the read-data mode reads every pack, and read-data-subset can divide that work into a weekly cycle. A real check looks like this:

# Проверка выбранной базовой копии PostgreSQL
pg_verifybackup /srv/backups/orders/base
status=$?
printf 'backup_integrity_status=%s\n' "$status"
exit "$status"

A successful full pass ends with no errors were found and exit code 0. Monitoring should receive the exit code, the age of the most recently read recovery point, the proportion of the volume checked, and the repository identifier. Searching a log for the word error is unreliable because message formats change and a warning may go unnoticed.

For PostgreSQL, the command pg_verifybackup /path/to/backup checks a base backup in plain format. It verifies file presence and checksums against backup_manifest, then checks the required WAL range. Run the pg_verifybackup version that matches the server version because WAL validation is version dependent. You still need to start the restored database afterward.

Let the backup product or database create checksums when it has that facility. A homemade hash file next to an archive helps only when the hash file itself has independent protection. If an attacker or faulty process can rewrite both the archive and the hash list with one account, the check will approve the altered state.

Do not automatically repair a repository after a check fails. Repair commands may change metadata or remove references to unavailable data. Save the log first, stop cleanup and rotation, copy the metadata, determine the cause, and then repair it under a documented procedure. Otherwise, a failure overnight can leave you with a tidy but incomplete set of recovery points in the morning.

An isolated environment protects production from the test

Run a trial recovery in a network where the restored system cannot reach production addresses. A copy of an application remembers schedules, SMTP addresses, queues, webhooks, exchange jobs, and credentials. If started normally, it may send old invoices, consume messages from a live queue, or begin replication in the wrong direction.

Isolation means a separate segment with no route into production subnets, not merely a different virtual machine name. Allow outbound connections only from an allowlist to dependency simulators, a package repository, and the result collection system. DNS in the environment should return test addresses. Replace email, SMS, payment gateways, and external APIs with stubs that retain requests for inspection but send nothing outside.

Microsoft's Azure Site Recovery test failover guide recommends selecting a network isolated from the production recovery site while repeating the structure of production subnets. The advice applies beyond one cloud: network logic should be similar, while connectivity to actual production should be absent. An oversimplified environment hides dependencies, while a fully connected one creates an incident during the exercise.

Credentials also need isolation. Do not give a restored machine a valid machine certificate or token when a test secret will support the check. If data cannot be decrypted without a production key, release it to the recovery process through a separate procedure, record the access, and revoke the temporary permission after completion. Never store the only copy of the key inside the system that must be recovered.

Check environment capacity in advance. Too little memory can cause a slow start that people misread as backup corruption. Too little disk space stops extraction near the end. Performance testing requires resources comparable with the target recovery platform. A smaller environment is acceptable for functional testing, but record that limitation and do not present the measured time as a confirmed RTO.

A trial recovery must start from empty resources

A clean recovery reproduces the work of an on-call team without hidden preparation. If the environment has kept an installed database, configured drivers, and an old configuration copy for months, the test checks an update to an existing environment rather than recovery after losing the system.

A practical run has five stages:

1. The on-call operator receives the scenario number, service name, target point, and time allowance, but not a prepared command containing the snapshot identifier.
2. Automation creates clean compute resources, an isolated network, empty disks, and stubs for external services.
3. The operator finds the required point in the catalog, obtains the key through the emergency procedure, and restores the data by following the current runbook.
4. The team starts dependencies in the correct order, applies only preapproved address changes, and runs application tests.
5. The service owner accepts or rejects the result, after which the environment is removed and temporary secrets are revoked.

Measure several timestamps rather than one overall duration: scenario declaration, access granted, reading started, data recovery completed, first successful start, test completion, and owner decision. A missed deadline then stops being abstract. You can see that 70 minutes were spent finding a password or creating a network, not reading disks.

Select the recovery point according to the scenario rather than always using the latest one. The latest backup checks routine recovery. A random point within the retention period checks the catalog and older media. A point just before logical corruption checks point-in-time recovery. Once a year, use a scenario in which the primary management server or site is unavailable, or the team will prove only the easiest path.

Do not allow the environment to write to the source repository when read access is sufficient for recovery. A separate read-only account reduces the chance that a script error will start cleanup or alter the catalog. Store test logs and acceptance artifacts elsewhere so that the evidence does not disappear with the system under test.

Application checks prove that the service works

An operating system boot is weaker evidence than it appears. It confirms the image and bootloader but says nothing about database completeness, transaction consistency, or a user's ability to complete the main task. Acceptance criteria should describe the function in language the service owner understands.

For a database, begin with technical checks: the server accepts connections, log replay ended without fatal errors, required schemas and extensions exist, and background processes run. Then test meaning. Compare previously recorded control values: the number of closed orders at the control point, the latest finalized document date, a total over a small immutable sample, and the presence of several known identifiers.

Keep control queries with the procedure but outside the backup. An example for a PostgreSQL application might look like this:

SELECT max(committed_at) AS newest_commit FROM orders;
SELECT status, count(*) FROM orders GROUP BY status ORDER BY status;
SELECT count(*) AS broken_refs
FROM order_items i LEFT JOIN orders o ON o.id = i.order_id
WHERE o.id IS NULL;

The expected result cannot be merely "the query ran." The first date must fall within the allowed RPO, the status distribution must be compared with the control snapshot, and broken_refs must equal zero. Choose queries that test the invariants of the specific system, or the test database may be available and useless at the same time.

For a file share, check the directory tree, permissions, owners, several files of different sizes and types, and whether an application can open a file. For a virtual machine, check network configuration, system time, service startup, and the absence of unexpected outbound requests. For an application with federated sign-in, provide a test identity. Without one, teams often declare success after viewing the login page even though no user can sign in.

Encryption and compression add separate failure classes. An archive may be intact while its key is lost. The key may exist while the emergency team lacks permission to obtain it. Recovery software may not support an old format. The run must therefore use the actual key access process and the software version from the emergency kit. A password remembered personally by an administrator is not a procedure.

Frequency depends on risk and rate of change

One quarterly schedule for every system is convenient for a calendar but poorly matched to risk. Trial recovery frequency should depend on allowed downtime, the rate of change, dependency complexity, and the cost of failure. Automated job and freshness checks normally run after every backup cycle. Plan full data reading so that the entire repository is covered within a known period.

For a service with a short RTO and frequent changes, a monthly automated recovery and a quarterly exercise involving people are reasonable. For a stable internal system that can remain down for several days, a quarterly run and one full annual scenario may suffice. These intervals are not a universal standard. The owner should justify them through risk, not through an empty slot in the team's calendar.

Some events should trigger a test without waiting for the calendar:

• a change of backup product, format, encryption, or retention policy;
• a major application, database, hypervisor, or network design upgrade;
• a repository move, key rotation, or change to emergency roles;
• a failed job, damaged medium, or recovery during a real incident;
• a change to RPO, RTO, service scope, or an external dependency.

Partial tests are useful between full runs. You can check freshness daily, read one repository portion weekly, restore a database automatically each month, and once a quarter give the run to the on-call shift without help from the procedure's author. Rotate the sample: restoring the same small directory every time produces precise knowledge about one directory and almost none about the rest of the data.

Plan test resources as normal workload. A full read competes with new backup writes and may increase data retrieval charges. Limit throughput, choose a window, and watch storage latency, but do not disable deep checking permanently because of its cost. If the organization cannot regularly read its own backup set, record that architectural property as a risk and change the storage design or test cycle.

A record turns a run into evidence

The result should answer five questions: what was recovered, from which point, who did it and with which procedure version, how long each stage took, and which checks passed. A screenshot of the final screen answers almost none of them.

A minimal record can be stored as JSON and sent automatically to the change log:

{
  "test_id": "restore-2026-04-orders-01",
  "service": "orders",
  "recovery_point_utc": "2026-04-14T01:30:00Z",
  "started_utc": "2026-04-14T03:00:00Z",
  "service_ready_utc": "2026-04-14T04:42:00Z",
  "rpo_minutes": 30,
  "rto_minutes": 240,
  "integrity_check": "passed",
  "application_checks": {"passed": 8, "failed": 0},
  "runbook_version": "git:7c91e2a",
  "operator": "oncall-a",
  "owner_decision": "accepted"
}

Attach the raw verification logs, backup and environment identifiers, control query results, deviation list, and owner decision. The record must contain no secrets. If a log contains a token or connection string, sanitize it before publishing it in the tracking system while preserving evidence of the timestamp and exit code.

Use at least four statuses: passed, failed, passed with limitation, and not run. The last one must not become successful merely because the backup exists. Describe a limitation precisely as well. "Functions checked at half production capacity, RTO not confirmed" says more than a yellow icon.

Every deviation needs an owner, a deadline, and a retest condition. Fix an instruction error by changing the instruction and rerunning the failed step. A capacity shortage requires a platform correction and a full recovery. If a team closes a finding with "we will consider it next time," the record becomes an archive of excuses.

Keep a trend of stage durations. A rise in read time with the same data volume may warn of storage or link degradation. More time before recovery starts usually points to access rights and a stale runbook. One total metric hides both cases.

Most failures occur next to the backup

A damaged archive does occur, but exercises more often fail in the surrounding environment. The catalog sees the point, but the recovery account has expired. The data decrypts, but the application certificate is missing. The database starts, but the message queue still points to a production address. A daily job report catches none of these causes.

The first common failure is letting backups and keys depend on one management domain. After that domain is compromised, an operator loses both the service and recovery access. Use separate emergency roles, a tested release procedure, and at least one route that does not require the primary identity system to work.

The second failure is a retention policy that looks sufficient by recovery point count but does not cover the time required to discover logical corruption. If an incorrect deletion is found after five weeks and usable points live for four, flawless recovery returns already damaged data. Compare retention with discovery time, legal requirements, and the availability of logs for point-in-time recovery.

The third failure is treating deduplication as independent copies. Several snapshots may reference common blocks, so losing one pack can affect many dates. That is normal for the technology, but it makes full reads and an independent copy of critical data especially significant. The number of snapshots does not show the number of independent copies.

The fourth failure is having the runbook author perform the exercise in a familiar environment. That person fills gaps from memory and misses ambiguities. At least periodically, an on-call operator who did not build the system should perform the procedure. Even a quiet question in chat counts as a runbook defect if the answer takes only a minute.

The fifth failure is ending recovery before application acceptance. The infrastructure team sees a running virtual machine, the application owner learns about the test a week later, and the record has already been signed. Assign the acceptance owner before the exercise and name a deputy. Without the owner's decision, the status remains technical rather than final.

An exercise ends only after the cause is fixed

A sound test program creates a repeatable chain: automation reads the repository, a clean environment restores the selected point, the application passes meaningful tests, the owner accepts the function, and deviations feed back into the runbook and architecture. Any missing link leaves a blind spot.

Describe the test environment as code and recreate it for every run. This reduces hidden manual settings and also tests whether the emergency platform can be deployed at all. Store that code outside the system being recovered, and test its changes with the same unscheduled run.

For organizations that need a separate recovery environment, GSE.kz can assemble server and data center infrastructure and perform system integration around the existing software stack. Acceptance criteria, emergency roles, and the discipline of repeated exercises still remain the service owner's responsibility.

After a failure, do not lower the difficulty of the next test to restore a green metric. Keep the original scenario, fix the specific cause, and repeat the whole exercise if the defect could have affected other stages. A backup becomes a working recovery mechanism only after that run, and the date of the latest accepted test says more about its reliability than the number of successful nightly jobs.