A backup that actually works: the 3-2-1 rule and where backups fail

Ask whoever runs your network two questions, and watch how fast the answer comes: if the accounting server stopped this minute and never came back, how many working hours do we lose? And how many days are we down? If the answer is hesitant, or a rough guess, what you have is not a backup in any useful sense — it is a job that runs overnight and a hope that it works.

A copy whose restore has never been tested is not a backup; it is an assumption. And the worst day to discover the assumption was wrong is the day you need it.

The two numbers that come before any purchase decision

Before any talk of disks, media and software, two numbers decide everything that follows. Neither is a technical decision; both belong to the business owner:

• How much data can you afford to lose? This is what RPO means. If the backup runs once, at eleven at night, you have quietly accepted losing a full working day. For an engineering office that may be acceptable. For a retail chain recording sales from morning until midnight, losing a day of sales is not — and the answer there is frequent database backups through the day, not a single nightly copy.
• How many days can you afford to be down? This is what RTO means, and it is the one number that is most often misjudged. The backup is on an external disk? Good. But restoring means sourcing replacement hardware, rebuilding the disk array, installing the operating system, and only then actually moving the data. And moving several terabytes over a gigabit network — which in practice gives you something close to a hundred megabytes per second — is measured in hours, not minutes, with the preparation hours added on top.

The relationship between them is simple and decisive: the smaller you want both numbers, the higher the cost. Hourly backups need more capacity and more bandwidth; recovery in hours rather than days needs spare hardware standing ready, not bought on the day of the incident. The mistake is not choosing a large number — it is not knowing which number you are already living with.

The 3-2-1 rule, and what has been added to it

The rule is older than the current wave of ransomware, and it still holds:

• 3 copies of the data at minimum — the original and two more.
• 2 different storage media, so that one failure of one type cannot take everything down.
• 1 copy off site. Fire, a water leak and theft do not distinguish between the server and the storage unit next to it in the same rack.

Two clauses were added later: at least one copy that is immutable or physically disconnected, and zero errors in a real restore test. Both were added for one reason: attacks now go after the backup before they go after the data.

And one point that always gets confused: RAID is not a backup

A RAID array inside the server protects you against a disk failure. It does not protect you against a folder deleted by mistake, against a corrupted database, or against every file being encrypted — all of which are written to the array immediately and with perfect fidelity. The same goes for snapshots on a storage unit such as the HPE MSA 2062: an excellent tool for stepping back minutes or hours, but they live on the same array in the same room, and they depend on the integrity of the very pool they are protecting. They are a fast recovery layer, not the copy that survives.

What ransomware does to a backup share it can reach

This is where the plan falls apart in practice. The company backs up on schedule to a network storage device in the same room, over an open share or with credentials saved on the server itself. From the attacker's point of view, that is not a backup — it is another folder on the network.

What actually happens once an attacker has privileges on the server:

• They enumerate the mapped drives and network shares available to the account they have compromised, and encrypt what is on them along with everything else.
• They hunt for backup files by their well-known extensions such as .vbk, .bak and .vhdx, and encrypt or delete those first — because removing the way back is what makes paying an option.
• They delete the Volume Shadow Copies on Windows with a single command, and with them goes the whole idea of "restore a previous version" from inside the system.
• They inherit the backup credentials saved on the server. And if the backup account is a member of Domain Admins — which happens because it is easier to set up that way — the attacker now holds the key to everything, the backup storage included.

The conclusion is meant literally: any copy that a compromised account inside your network can delete or write over is not a backup. That makes securing the ways into the network part of the backup plan rather than a separate subject — and one of those recurring ways in is remote access left open to the internet.

How to make a copy the attacker cannot reach

There are three real ways to achieve that separation, and any one of them you actually implement is better than none:

1. An immutable copy

Storage that will not let a file be changed or deleted before a set period has passed. You get this through Object Lock in S3-compatible cloud storage, or through a hardened Linux repository where the backup software sets the immutable flag for a period you choose.

And one detail here is worth attention when you are buying: Object Lock has two modes. Governance blocks deletion for ordinary users, but anyone holding the permission to bypass it can lift the lock and delete — meaning it protects you against mistakes more than against an attacker who has reached an administrative account. Compliance blocks deletion, and any shortening of the period, for everyone including the root account holder, for the whole retention period. If the purpose is protection against ransomware, the second mode is the one you mean — and be aware that the capacity stays reserved and billed until the period expires. That is the price of the guarantee, not a flaw in it.

2. Physical separation

The simplest method, the cheapest, and the most neglected: an external disk disconnected once the backup finishes, or an LTO tape taken out of the drive. Media that is not connected cannot be encrypted remotely, whatever privileges the attacker has. In practice you rotate two disks, one of them always out of the building. Its weakness is that it depends on human discipline about the swap, which is where a named person and a written log come in, rather than "whoever remembers".

3. A separate account and remote storage

A copy at another site or in the cloud, under an account not tied to the company domain, with a completely different password and a second factor. An attacker who owns the domain owns everything that authenticates against the domain.

And the warning that has to go with it: synchronisation is not backup. A folder that syncs automatically will upload the encrypted files over the good ones with perfect efficiency. It is true that services such as OneDrive for Business provide a bulk rollback to an earlier point in time, but its window is thirty days, and the recycle bin runs to about ninety-three days by default — and then nothing. It also covers synced folders only: not the server, not the point-of-sale database, not the virtual machines. Treat it as an extra layer, not a plan.

Testing: the part that gets postponed until it is too late

Backups fail in dull, repetitive ways, not dramatic ones:

• The job has been stopped for months and the alert goes to the mailbox of an employee who has left. Nobody noticed, because success is silent and so is failure.
• The backup exists, but it is inconsistent. A database copied as a file while it was open, instead of using its own backup tool or a mechanism that guarantees a consistent point in time, comes out unopenable. It looks healthy in the report, and fails at restore.
• The backup is encrypted and nobody knows the encryption password since the person who set it up left.
• The scope is incomplete. The server is backed up, but the firewall configuration, the point-of-sale database and the file shares on another machine are not — and nobody noticed, because the list was never written down.

The test that exposes all of the above is simple: restore a single file every month, and restore a full system into a test environment every three months. Record the time in hours, from the moment you start to the moment the system is genuinely working. That figure — not the optimistic estimate — is your real recovery time. Alongside it, write a short restore document and keep a copy printed or off the network, because an instructions file saved on the encrypted server will do you no good.

Retention: why the latest backup is not enough

If you keep one copy that is overwritten every night, you are protecting yourself against sudden failure and nothing else. Silent corruption in a database, encryption that started in the background before announcing itself, a file deleted without anyone noticing it was gone — all of these come to light days later. And if the only copy has been overwritten in the meantime, you have written over the last good backup with your own hands.

A practical, balanced pattern: daily backups kept for two weeks, weekly kept for two months, monthly kept for a year. That gives you genuine depth in time at a reasonable cost in storage. And retention is not purely a technical matter: holding customer data indefinitely widens the damage any breach can do, and it is also a subject with a legal side in Egypt that ties how long you keep data to the purpose you are processing it for.

What this means on the ground

Two patterns recur in small and medium-sized companies. The first: one capable server — an HPE ProLiant DL380 Gen11 or its equivalent — running everything. Here the first rule is that the backup is not written to that same server, however separate the disk may look; you need a second medium and a destination outside the building.

The second: several branches connected to a head office — the structure whose network, racks and cabling we built out across the Al Raya Market branches and head office. In a setup like that, the real design decision is where the first copy is kept: restoring from a local copy at the branch is far faster, while pushing everything up to head office overnight may be more than the branch line can carry. The practical answer is usually a local copy for speed, plus a consolidated copy of the critical data only held outside the branch — and that requires knowing which data is critical, which is a list only the person running the business can draw up.

In both cases, preparing the room itself is part of the picture: power, cooling, and a rack layout that makes it easy to disconnect a medium and carry it out. At ZEMAVO we supply and install servers and storage units, and monitoring backups and testing restores is a line item within our support and maintenance contracts — the line that is first to be cut from a quotation and first to be missed when something goes wrong.

Six questions to answer this week

• Which systems are actually being backed up? Ask for the list in writing and compare it with what the company cannot operate without.
• When was the last successful backup? Where does the failure alert go, and does the person receiving it still work here?
• Where is the off-site copy, and when did it last leave the building?
• Could a compromised account inside the network delete that copy or write over it?
• When was the last tested restore, and how many hours did it take?
• What are the written numbers for RPO and RTO, and who approved them?

Backup is not a product you buy once; it is a procedure you test. On the day it matters, the difference is not between who spent more and who spent less, but between whoever has restored before and whoever is doing it for the first time with the business at a standstill. If you want your current position reviewed, get in touch and we will start from the written list.