Monday, July 19, 2010

Backup, Backup Backup - never underestimate the value of it.

There I was, bitching about non-geek stuff, when *poof*, suffice it to say, never underestimate the value of a good backup strategy.

One of the Moodle virtual servers that I manage failed and (conveniently) the RAID 5 with the backups went up in smoke as well. It literally just disappeared. Usually I prefer Hard disk backup to tape but in this case, hosting a server for someone else (bureaucratic favour), my preference turned around and bit me in the ass.

The meta-data on the Raid controller apparently became corrupt, with some sort of power blip (even with enterprise level UPS and generator backups) that happened just at the wrong time possible. Kind of like lightning hitting you in the dead of summer whilst you were casually walking in a field of lightning rods.

Anyhow, disaster aside, time to attempt either a recovery or a rebuild.

Part One - RAID Recovery.
In order to get the server to recognize the drives once more, the array needed to be deleted. Sounds scary, but technically the drives will not be wiped, per se. Then the array needed to be recreated using the same settings as its predecessor. Blind luck helped in this case, as the original guy who created the array took off, presumably because he got a premonition about this shitstorm and decided to flee.

So the Array got recreated and the server was able to see the 500 GB once again. YAY!...... well no. The 500GB appeared to be empty.

Time to see if we can work some magic.

Part two -  Attempted data recovery.
This will be done using UBCD for linux - two words. GET IT.

Using the PhotoRec tool by CG Security, which is included on the CD, an attempt is currently being made to recover the data.

An update will follow soon.


All we managed to recover was junk data. The server was rebuilt using an older backup. The client now understands the importance of a little request, like an extra HDD for backups.