How do you keep an off-site backup over a slow link?

An off-site backup works over a slow link if you do not try to push the entire original volume through it. Move the complete copy once on encrypted media, then let the link carry only new blocks. This is not a temporary concession to poor connectivity. It is a normal design for branches, industrial sites, and smaller towns.

The weak point in this design is rarely the speed of the nightly job. Teams celebrate a green transfer status without knowing whether the remote copy matches the source, whether it will survive an accidental deletion, or how long several terabytes will take to bring back. Design around recovery instead: how much data you will need to return, where it will come from, and how you will deliver it if the main link is busy or down.

Put RPO and RTO into measurable numbers first

A slow link is suitable for backup when the daily change volume fits into the available window and the return method meets the allowed downtime. Total file size says very little by itself. A 20 TB store that changes by 40 GB a day is easier to protect over a 20 Mbit/s link than a 2 TB database that rewrites most of its blocks every day.

RPO answers how much of the latest data the organization is prepared to lose. If backup runs once a day, the actual RPO can approach 24 hours plus the duration of a failed job until the next successful run. RTO defines the allowed time before the service returns, not the time before a download starts. It includes fault detection, the recovery decision, access to keys, hardware preparation, retrieval from a cold tier, delivery, reading, decryption, verification, and application startup.

Collect measurements for at least two ordinary weeks and separately for month-end closing, a large update, or a seasonal peak. You need four figures: the logical protected data volume, bytes actually transferred after compression and deduplication, job duration, and the lowest available throughput in the backup window. Average link speed is misleading. Backup competes with telephony, tills, video calls, and business traffic, so calculate with the guaranteed remainder.

A quick throughput check looks like this:

полезные_ГБ_за_окно = Мбит_с × 3600 × часы × коэффициент_канала / 8000

The link factor accounts for protocol overhead, retransmissions, and fluctuation. I use 0.7 for a preliminary calculation, then replace it with the result of a controlled transfer. With 10 Mbit/s and an eight-hour window, the result is about 25.2 GB: 10 × 3600 × 8 × 0,7 / 8000. If the site changes 35 GB overnight, the queue will grow by roughly 10 GB every day. Compression will not fix this consistently when the data is already compressed or encrypted.

Do not confuse snapshot frequency with the RPO of the remote copy. A local snapshot every 15 minutes does nothing during an office fire if the last confirmed state outside the building dates from the previous evening. Measure the age of the latest local snapshot and the latest verified remote restore point separately.

Treat the initial copy as a controlled shipment

Create the initial seed on removable media next to the source, physically deliver it to the remote site, and import it into a prepared repository. Trying to send the original terabytes over a weak link for months creates false confidence: until the queue reaches zero, the remote side does not hold a complete restore point.

Encrypt the media before writing and send the key by another route. A password on a note in the same case turns encryption into decoration. Use the repository's or container's built-in encryption with an algorithm and format your team can recover on a clean machine. Keep the key in a secret manager or a sealed emergency kit that at least two appointed staff members can access.

Create the initial copy from a consistent snapshot, not a live directory that changes during hours of reading. Virtual machines need an application-aware snapshot. A database needs its supported backup format or a coordinated volume snapshot with logs. A copy of open database files may pass every hash check and still refuse to start.

I use a short chain-of-custody record even when an employee carries the disk:

1. Record the media identifier and serial number without the secret key.
2. Note the snapshot, its creation time, size, and repository control identifier.
3. The sender and receiver record transfer time and seal condition.
4. After import, the receiver runs a full data check and a trial recovery.
5. Wipe the media under the approved procedure, or add it to a rotation only after confirming the import.

Do not ship the only local backup. The seed must create another copy, not move the existing one. Do not accept courier delivery without a chain of responsibility either. A lost encrypted disk usually does not expose the data, but poor records leave you unable to prove which set went out or whether anyone imported it.

After import, do not blindly start an incremental chain from a new snapshot. Make a small change at the source, transfer it over the normal link, restore the changed file from the remote side, and compare its contents. This test proves that the physical import and subsequent network transfer converged in the same repository.

Send changed blocks, not merely changed files

Source-side deduplication matters on a weak link: the program must identify blocks already held by the repository before transmission and send only missing ones. File-level incremental backup saves traffic when new documents appear. It barely helps with a large virtual disk or mail archive in which one percent changed if the program sends the entire file again.

People often blur three separate mechanisms. An incremental snapshot stores changes relative to a previous state. Deduplication does not send a block the repository already knows. Delta transfer compares a file held on both sides and sends the differing portions. These mechanisms can work together, but none guarantees another.

The rsync manual explicitly says its delta algorithm reduces traffic by sending differences between the source and destination file. It also describes the standard quick check based on file size and modification time. That is a useful synchronization shortcut, but rsync does not create a history of immutable restore points by itself. A command with --delete can faithfully carry an accidental deletion or ransomware activity to the remote side.

If rsync is part of the design, use it as transport to a versioned receiver, not as the sole backup. A dry run helps reveal the proposed changes:

rsync -a --dry-run --itemize-changes /data/ backup-host:/incoming/

A line such as >f.st...... reports/month.csv means that rsync will transfer a regular file with a changed modification time. However, matching size and time do not prove matching content under the default check. The --checksum option changes the initial comparison, but forces both sides to read the files and may put a heavy load on the disks. Use it for periodic reconciliation or suspicious data sets instead of enabling it everywhere without measurements.

For databases, prefer native full backups and logs, or use software that understands application consistency. For virtual machines, check whether the system reads only allocated blocks and tracks changed blocks after every snapshot. Block deduplication may almost disappear for an already encrypted container: a small plaintext change can alter a large region of ciphertext. In that case, split the data before repository encryption if the threat model permits it.

Throttle the stream so the queue keeps shrinking

Backup should yield to business traffic, but an excessively low hard limit can make lag permanent. Reserve bandwidth for essential services first, then let the job use the remainder through a schedule or quality-of-service priority. Raise the cap at night and lower it during the day instead of stopping transmission entirely. Frequent interruptions add repeated checks and lengthen the tail of a job.

Watch the queue of unconfirmed changes as well as the current run speed. A useful daily report has five fields: time of the latest successful remote point, bytes created, bytes transferred, bytes remaining, and expected time to clear the queue. If incoming changes exceed outgoing traffic for three consecutive days, raise an alert before the disk fills and the RPO fails.

Schedule by data class. Send transaction logs and small configuration files frequently. Images, media, and archives can travel at night. Exclude temporary files, caches, installation packages, and data that can be recreated reliably after agreement with the service owner. An exclusion must not start as an administrator's guess. One forgotten directory containing configuration keys can make a restored server useless.

Account for the overhead of small files. A million files of a few kilobytes can spend more time on directory traversal, metadata, and network latency than on useful bytes. Packing them into a supported backup format solves the problem better than a hand-made archive that changes in full on every run.

Another awkward case follows a long outage. The site reconnects with a large queue while continuing to create new changes. Do not give it the whole link. Temporarily extend the window, deliver an intermediate set on media, or increase the cap during agreed hours. If the calculation shows weeks of lag, a second physical seed usually costs less than waiting.

Check integrity at three different levels

A successful transfer proves only that the program completed without a known error. A usable copy needs three checks: the transport did not corrupt accepted objects, the repository structure is coherent, and the application can open restored data. One green tick cannot replace the others.

At the transport level, compare cryptographic hashes or use the program's built-in verification. The rclone documentation says a normal transfer compares available checksums and reports an error when it finds corruption. For encrypted remote storage, rclone cryptcheck compares the source with encrypted objects when the underlying storage supports a checksum. The limitation matters: checking names and sizes is not the same as reading every byte.

At repository level, check indexes, snapshots, references to blocks, and the data blocks themselves. The restic documentation makes a useful distinction: plain restic check verifies structure, while reading all data requires a separate option. An operating sequence can look like this:

restic check
restic check --read-data-subset=1/7

Run the first command often. The second reads one of seven parts. By changing the number every day, the team covers the whole repository in a week without loading the weak link all at once. A complete pass is also needed periodically after replacing storage, importing an initial set, or finding an error. Expected successful output includes stages for loading indexes and checking packs, snapshots, trees, and blocks, although exact lines depend on the version.

At application level, restore the selected point to an isolated location. Open documents, mount the file system, run the database vendor's verification tool, and start the service without connecting it to production queues. A hash of the source backup object will not detect a missing log, a forgotten certificate, or an incompatible server version.

Store verification results outside the protected site. A record should associate the snapshot, repository, command, program version, time, bytes read, and result. Do not turn the record into a dump of complete console output. A short machine-readable status with a reference to a locally stored detailed log makes alerting and auditing easier.

The remote copy must survive an administrator account

Distance in kilometres does not protect against ransomware, an automation error, or stolen credentials. If the backup server can delete every remote point with the same credentials it uses to write new ones, an attacker gets both capabilities at once.

Separate write, retention, and deletion roles. A site agent should add data but must not alter old objects or shorten their retention. Allow deletion through a separate account, an approval procedure, or an immutable retention policy at the receiver. An administrator of the source domain should not automatically own the remote store.

Versioning and immutability solve different problems. Versioning retains previous object variants after overwriting or deletion. An immutable retention period prevents deletion of a version before a set date, even by an authorized account within the selected mode. Check whether protection covers the repository's service indexes and metadata. Preserved blocks are useless if someone can destroy the restore-point catalogue and the team does not know how to rebuild it.

Base retention on detection time, not on a familiar seven-daily, four-weekly grid. If silent damage is found after 20 days, two weeks of history will not help. A longer history also increases cost and verification time. Agree daily, weekly, and monthly points with data owners, retention requirements, and the likely detection window.

Encryption of the remote copy is mandatory, but it creates a dependency on the key. Run a separate exercise with a clean machine, a documented repository address, emergency access to the secret, and a staff member who did not configure the system. If that person cannot unlock the copy without calling an administrator who has left the organization, the copy effectively does not exist.

Calculate recovery time from the narrowest stage

A recovery calculation starts with the selected restore point's volume, not the whole repository. For every stage, use an attainable rate: remote-store preparation, object release, the link, local writing, decryption, unpacking, and verification. Stages can sometimes overlap, but I add them first in a plan. Only a test can prove an optimistic parallel schedule.

Use this formula for transmission:

время_скачивания_час = объем_ГБ × 8000 / (Мбит_с × коэффициент_канала × 3600)

Restoring 4 TB over a usable 20 Mbit/s path with a 0.7 factor takes about 635 hours, more than 26 days. Even a 100 Mbit/s link cuts pure transfer time only to about 127 hours. These figures quickly reveal why data stored somewhere far away does not meet an eight-hour RTO.

Add fixed delays. A cold tier may need time to prepare objects. A provider or remote office may release media only during business hours. A courier depends on distance and weather. The site needs a server with enough capacity and write speed. After 4 TB arrives on a disk, local reading at 150 MB/s takes about 7.4 hours in theory, but many files, verification, and a slow array extend that time.

Plan two return paths. The first handles small incidents: individual files, a folder, one database, or a log travel over the network. The second handles loss of a site: the remote side exports the selected point to encrypted media, transfers it through the agreed chain, and the team restores it to standby hardware. An essential service may keep a minimal data set or replica nearby, but a replica does not replace historical backup.

Write down the switching threshold. For example, the team might recover up to 100 GB over the network and compare expected download time with disk preparation and delivery above that level. The threshold depends on the link and geography, so do not copy a number from someone else's runbook. During an incident, the decision should require two measured values, not a meeting lasting several hours.

A recovery test must break convenient assumptions

Run trial recovery from the remote copy with the access that will remain after the office is lost. NIST SP 800-34 includes restoration of selected functions from a sample of backup information in contingency-plan testing. I like the word restoration in that requirement: checking a file list or a successful-job report does not demonstrate the return of a service.

A good exercise begins with a restore point and an expected result. Select a date, a server, and a business test, such as opening a document, querying a database, and getting an agreed record count. Appoint an observer who records stage times but does not provide hints from personal memory. The team must work from the runbook and emergency secrets.

Test a randomly selected valid point rather than the newest snapshot. A recent copy is more likely to be in a fast tier and familiar to administrators. An older point exercises retention, compatibility, cold storage, and the log chain. Once a year, it is useful to declare the source site unavailable during the exercise and prohibit reading the local backup catalogue.

After the test, retain actual timings: access obtained, first byte restored, last byte restored, application verified, and service ready for users. Break any RTO gap down by stage. Buying a wider link helps only when the network is the bottleneck. It does not accelerate manual key approval or slow writing to an unprepared array.

Do not call an exercise successful if the application starts with unknown losses. Compare the restore point with the stated RPO, check the latest available transactions, and record excluded data sets. Then fix the runbook, permissions, capacity, or backup design and repeat the failed stage. A report without a retest merely describes the remaining risk.

Test recovery while the link is busy as a separate case. On an ordinary day, backup uses spare bandwidth. During an incident, users may simultaneously connect to the standby site, exchange files, and make calls over the same network. A night-time test download will show an attractive rate that disappears during working hours. Measure with an agreed load and record the minimum rate at which services remain usable.

Prepare a recovery destination in advance. A 20 TB repository does not mean every incident requires 20 TB of free space, but the target server must hold the selected point, temporary unpacking files, and working growth before cutover. Check the file system, permissions, hypervisor versions, controller drivers, and availability of installation packages without the original office. I have seen a sound copy wait hours for the right network driver on a standby server. That was not a backup failure, but the service owner saw no difference.

Separate tests by scale. Recover one random file and a small database monthly, bring up a complete service quarterly, and test the bulk scenario with media delivery after a substantial infrastructure change. Frequency depends on risk, but every test type needs an owner, a due date, and a success criterion. Automated file verification does not replace service startup, and a large annual test does not justify eleven months without small recoveries.

The runbook must list decision points. Who declares the incident, who approves access to the key, who selects the restore point, who orders the media, and who confirms application integrity? Record contacts by role and include a backup communication channel. Employee names become stale quickly, while a role without an assigned person on call is equally useless. Time each approval during an exercise because organizational delay often exceeds the technical operation.

Remember to test after updating the backup program itself. A new client may read an old format only after an index migration, while an emergency image with an outdated version may not recognize the new repository. Keep a supported installation package, its checksum, installation instructions, and compatibility information with the emergency documentation, but outside the only protected site. A licence and vendor account can also become dependencies worth testing in advance.

Finally, test the absence of one person. Let an on-call engineer who has seen the runbook but did not build the system perform the recovery. The project author observes and records every place where the text allows two interpretations, omits an exact command, or assumes an unknown path. The test can be uncomfortable, but it converts an administrator's personal skill into a repeatable procedure. If the result depends on one person's memory, a holiday or resignation can make a sound repository inaccessible.

The operating design should fit on one page

The operating design must be clear enough for an on-call engineer to understand its state without the project's author. On one page, list the sources, local staging point, remote repository, schedule, link caps, retention, key owners, checks, and both recovery paths. Keep beside it the measurements that support the RPO and RTO.

A minimal policy can look like this:

source_snapshot: every_4_hours
offsite_transfer: continuous_with_night_boost
bandwidth_limit_day_mbps: 4
bandwidth_limit_night_mbps: 18
max_offsite_age_hours: 8
integrity_structure_check: daily
integrity_data_fraction: 1/7_daily
restore_test: quarterly
bulk_restore_path: encrypted_courier_media

This is not a ready-made program configuration. It is an agreement among the service owner, networking, security, and operations. Values must come from measurements and tolerable loss. If the monitoring system cannot check max_offsite_age_hours, turn that measure into a separate alert instead of relying on a successful-job email.

For sites in Kazakhstan, physical delivery is especially useful where distance, the cost of guaranteed bandwidth, or an unstable link makes a full network upload impractical. GSE can design server-room and data-centre infrastructure, select local servers, and integrate the system without tying it to a single vendor. The customer must still state acceptable data loss and downtime because an integrator cannot derive those numbers from disk capacity.

Begin deployment with a controlled recovery of one selected service, not with a storage purchase. It quickly uncovers a forgotten key, an incomplete set, a slow disk, and an unrealistic RTO. The initial shipment, daily delta, and checks then become a measurable process, and the remote copy becomes a way to restore work instead of an expensive warehouse of encrypted files.