RAID Data Recovery Bracknell

More arrays are ruined by this than by any one of their components. Rebuilding causes all the members still alive after the failure to read through from beginning to end for hours as they write to the replacement, which is the heaviest load you could put on disks that have been in the same warm cupboard together for six years. One of the disks fails at about the middle point of the rebuild, and now there are two failures in the set, plus a half-written new disk. If everything is copied first, none of that can happen.

Somebody removes all the disks to read their serial numbers, and afterwards they can't tell which slot each one was removed from. For a volume to appear, member order, stripe width and where on the disk data actually starts (the offset) must all be in agreement, and if the order is wrong you get something that mounts but reads as nonsense when you attempt to access its contents. The order can be worked out from the contents themselves, but it takes an afternoon of arithmetic, and a little bit of effort with some masking tape could have prevented all of this.

The cache was holding writes, the backup battery had been flat since last year, and then the power went out. What came back had parity blocks that described stripes which were never committed. This gap in the data is the "write hole", and it does just what its name suggests. Both versions can be put side by side on copies, allowing the correct one to be selected. On the live array, the controller settles the argument by overwriting one of them without asking.

Hardware controllers store their array definitions in memory on the card, so when that card dies those definitions become unreadable. That's why nobody here goes hunting for the exact same controller model on an auction site. The stripe boundaries, the parity rotation and the beginning of the file system can all be read from what each member disk contains, and getting them back off the platters is quicker and more reliable than trying to source a replacement controller.

Adding a new disk to a running RAID 5 will rewrite every stripe on the set at a new width, in place, over a period of days. If that process is stopped before completion, the array is left maintaining both geometries at once: the older one behind the point it had got to, and the newer one in front of it. The boundary area shows clearly in the images, and from there each area can be read using its own geometry.

A disk which dropped out of the set in April and was pushed back into it in June will be offering the controller an outdated (two months old) version of the file system. The two versions are going to pull against each other along the line where they stop agreeing. Pulling this seam apart is done on copies of the disks, as opposed to the originals, because on a copy a bad assumption costs one hour rather than a whole directory tree.

All of these programs (chkdsk, fsck and the vendor repair utilities) assume that the volume below them is basically sound. Run one against an array that has come back together in the wrong order and you'll find that it will go ahead and rewrite the metadata to match a layout that never existed. That is a second job created on top of the first, and that's why nothing is repaired here until the geometry has been proved.

Somebody launches the controller utility to check what's going on, works through the wizard, and creates a new array on hardware that already had one. Initialisation writes from the front, so if it was stopped within the first few minutes, the majority of the previous volume will still be lying there behind it, untouched, and it can be read perfectly well once the old geometry has been worked out.

The death of a PSU or a single spike on the rail can take out several members at once, and no level of redundancy is designed to protect against that. Each casualty is treated separately as a drive recovery in its own right on the mechanical bench before any of them is asked to contribute to a reconstruction, and the arithmetic across the whole group comes last, not first.

If an entire column of bays drops out at once, then six disks failing at the same time is the least probable explanation. Most of these come down to chassis wiring, a tired expander board or a cracked solder joint on the backplane. Finding out which it is costs nothing but an hour, and that is much cheaper than treating six good disks as casualties.

The spare sits, powered on but idle, for four years. At two in the morning the array loses a member, and the automated rebuild starts onto a disk that hasn't done one thing since installation. Idle doesn't mean healthy. Nothing here joins a reconstruction before it has been imaged and read clean, and a spare does not receive an exception to that.

The controller offers to import some metadata it has found, someone clicks "yes" because the other box looked worse, and the order of members changes underneath a working file system. However, the genuine layout is still written across the drives, and the drives are more truthful about themselves than a card that's just been given stale information.

Every once in a while the geometry comes out cleanly and all of the parity checks match up, but the file system on top (NTFS or ext4) is still wrecked. So there are two jobs stacked here: an array recovery underneath a file-system recovery. The second part is finished by reading the structures directly instead of pointing a downloaded scanner at the result.

A disk that has bad heads will go to the head-swap bench for a head transplant, have its firmware brought back into line, and only then join the imaging queue with all of the others. The other disks wait, since a set of disks is only as recoverable as the worst disk within it, and nothing is gained by starting the reconstruction early.

A small business array is virtually always made from a single order of identical disk drives. These all run the same hours at the same temperatures in the same box, so they reach the end of their working lives within weeks of each other, and the failure that gets noticed is rarely the first one. That is exactly why every member is imaged in parallel here.

The mdadm superblocks vanish after a distribution upgrade. Storage Spaces loses track of a pool that it created itself. It does not take much provoking for either to happen, and neither is as well documented as the wiki page suggests. Both are rebuilt by hand from the raw members, which is obviously slower than any automated route, but much more likely to work.

A disk with sectors being remapped on a schedule is not failing suddenly, it is failing predictably. When it's imaged while there is still slack in that schedule, the whole set comes home intact. If it is left running in service until the count runs out, some of it won't, and the part that is lost goes at random instead of being chosen by importance.

According to the design document the set is finished, since losing an entire mirror pair puts a hole straight through the stripe. In practice one of these two disks will usually image far enough on lab hardware to fill that gap, and the recovered volume is either complete or very nearly so. This is where the theory and the bench work tend to part company.

Consumer-grade (desktop) drives will spend a minute or more trying to read a single marginal sector, and on a desktop that is the right behaviour. A RAID controller waits seven seconds and then throws the disk out of the set for insubordination. On imaging hardware that is happy to wait all night, those same drives read perfectly well and the set goes back together.

Buy the cheap high-capacity drive, place it into a degraded array, and the rebuild will slow to a crawl as the drive shuffles overlapping tracks about behind the controller. Eventually it stops answering, the controller drops it from the array, and the array is in a worse state than before it was touched. It is worth knowing which disks were shingled before anything is ordered.

Virtualisation hosts running ZFS arrive after a power cut with pool metadata that was halfway through an update. Full images of every member come first (without exception), and then the transaction history is walked backwards on those copies until a consistent point is found. Typically, this point is only seconds behind the failure.

A patrol read or a monthly scrub means hours of uninterrupted reading across every member, and that is exactly what a marginal disk can't survive. The failure happens somewhere part way through the scan, at a weekend, and no one sees the email. If a set is going to be imaged, image it between the checks instead of after one.

An array that is short of a member has no way to correct a medium error, so it records the failure into parity as a permanent hole. Those punctures are mapped from the images and routed around at file level, which usually means a handful of files are affected rather than the whole volume. Nobody finds out until the rebuild reaches them.

Sector formats don't mix, so the controller rejects the newcomer, and forcing the point just creates two more problems on the path to fixing nothing. On images the sector size can be adjusted at will, a flexibility that never exists on hardware that is trying to be helpful.

HBA and expander updates that fail to complete will shed disks in ones and twos over the following fortnight. The console reports it as a series of drive failures, however this is far from what is happening. Working out whether the fault lies on the controller side or the media side comes long before anyone begins replacing disks that are perfectly good.

A stripe set with no parity anywhere in it, which was chosen because the numbers looked good in a review. At the exact moment one module stops answering, the whole volume goes with it. Each module is handled independently as an NVMe job on its own terms, and the stripe is then reassembled from the images afterwards.

Warnings had been going to the mailbox of a staff member who left before Easter, so for months nobody knew the set was running unprotected, and by the time someone rings, the second disk has gone as well. Everything in the chassis gets copied here, even the disks the controller still calls healthy, because a controller that missed the first failure is no judge of the rest.

Nominally identical capacities can vary between production runs by as many as thousands of sectors. This means the controller refuses the new disk, and somebody clears the configuration to make it accept one. At this point there are two faults, with the original fault still present but now accompanied by another. Because copies can be made to whatever size the work needs, the capacity argument simply does not arise on images.

Drives are removed from a decommissioned server and installed in a new one, and they still carry the metadata of the array they were part of previously. A controller that can read both sets of signatures will start making decisions nobody asked it to make. The layout we care about is settled from what data is stored on the platters, not by whichever signature made the most noise.

JBOD spans contain no form of redundancy at all. Files are laid down across the members one after another, so a missing disk removes a slice from the middle of the file system rather than cutting off the end of it. Everything that happened to sit entirely on the remaining disks comes back unaltered, and on a large span that represents a much larger percentage than most owners expect to hear.