sysop@hub:~/explainers/war-stories/ransomware-backup-gap$
The backup that had never been tested
Backups were running every night for three years. Nobody had actually tried to restore from one until the night it mattered.
timeline
Monitoring alerts fire across multiple servers within a few minutes of each other: file shares becoming unreadable, unusual process activity, and a spike in file rename operations consistent with encryption.
The on-call admin isolates the affected servers from the network to stop the spread, following the incident response plan, and wakes the rest of the team.
The team confirms ransomware has encrypted the primary file server and two application servers. A ransom note references a specific extension added to every encrypted file.
The team begins the recovery process: restore from last night's backup onto clean, reimaged servers rather than paying or negotiating.
The restore job for the file server fails partway through with a corruption error. The backup software has been silently failing on this particular job for the past six weeks after a storage change, but the failure emails were being filtered into a folder nobody checked.
An older backup, from two weeks before the storage change, restores successfully. Six weeks of file changes are permanently lost.
Application servers restore without issue, since their backup jobs were unaffected. Business operations resume with data current up to the older restore point.
A review finds the ransomware's initial access was a compromised VPN account without MFA, unrelated to the backup failure, but the backup gap turned a bad day into a much worse one.
- The backup job for the file server had been failing silently for six weeks. Alerts existed but nobody was monitoring the mailbox they were sent to.
- No one had performed a full restore test in the year before the incident, so the failure was discovered at the worst possible time.
- The successful application server backups worked because a different admin happened to test those manually a month earlier for an unrelated reason, not because of a formal testing process.
- Test a full restore, not just a file-level restore, at least twice a year, and treat a failed test as an incident in its own right. Backup and DR checklist
- Send backup failure alerts somewhere that is actually monitored daily, and consider a second, independent check that a backup job actually completed.
- Keep at least one backup copy offline or immutable, since it limits how much damage a live attacker or a live software fault can do to your last good copy. Glossary: network segmentation
- Require MFA on all remote access including VPN, since this is where the incident actually started. Entra ID hardening checklist
This is a composite scenario built from common, publicly documented attack patterns. It does not describe a specific real organization or incident.