Backup, Backup Backup - never underestimate the value of it.

There I was, bitching about non-geek stuff, when *poof*, suffice it to say, never underestimate the value of a good backup strategy.


Situation:
One of the Moodle virtual servers that I manage failed and (conveniently) the RAID 5 with the backups went up in smoke as well. It literally just disappeared. Usually I prefer Hard disk backup to tape but in this case, hosting a server for someone else (bureaucratic favour), my preference turned around and bit me in the ass.

The meta-data on the Raid controller apparently became corrupt, with some sort of power blip (even with enterprise level UPS and generator backups) that happened just at the wrong time possible. Kind of like lightning hitting you in the dead of summer whilst you were casually walking in a field of lightning rods.

Anyhow, disaster aside, time to attempt either a recovery or a rebuild.

Part One - RAID Recovery.
In order to get the server to recognize the drives once more, the array needed to be deleted. Sounds scary, but technically the drives will not be wiped, per se. Then the array needed to be recreated using the same settings as its predecessor. Blind luck helped in this case, as the original guy who created the array took off, presumably because he got a premonition about this shitstorm and decided to flee.

So the Array got recreated and the server was able to see the 500 GB once again. YAY!...... well no. The 500GB appeared to be empty.

Time to see if we can work some magic.

Part two -  Attempted data recovery.
This will be done using UBCD for linux - two words. GET IT.
http://www.ultimatebootcd.com/download.html

Using the PhotoRec tool by CG Security, which is included on the CD, an attempt is currently being made to recover the data.

An update will follow soon.


*Update*

All we managed to recover was junk data. The server was rebuilt using an older backup. The client now understands the importance of a little request, like an extra HDD for backups.

-n

Backing up Moodle Datafiles to a remote server

 As part of any disaster recovery plan, backup of the MoodleData folder is also critical. The critical variable in this equation is Disk Space - there never seems to be enough!

Making regular backups of the entire MoodleData folder is a bit impractical - at least in my scenario, where it is over 100 GB. Instead, I chose to mirror the moodledata folder on a different server, using rsync. This is purely for a disaster recovery scenario and will not consider data recovery on a per-user basis. In an environment of 22000+ users, this is highly impractical.

Assumptions: Another server has already been configured with CentOS, all necessary firewall/SELinux/networking settings have been completed.
In this example, I will use:
192.168.100.1 as the Moodle Server
192.168.100.2 as the Backup Server


0. Login as root on Moodle Server

A "trust" needs to be setup between the Moodle Server and the Backup Server, so that the automated backup does not fail due to password prompts! This is also an added measure of security as well, so that root / sudo passwords are not stored in plain-text in the shell scripts.


1. Setup a trust for password free rsync over ssh
ssh-keygen -t dsa
cat ~/.ssh/id_dsa.pub | ssh root@192.168.100.2 'cat >> .ssh/authorized_keys'
Prompted for the password ONCE. Enter it correctly and that's it!


2. Test the password free connection (this will only be one way). From Moodle Server:
ssh -l root 192.168.100.2
The backup server should be accessible without entering a password!


3. On the Moodle Backup, create a folder on the largest available disk.
mkdir /path/to/mirror

4. On the Moodle Server, create the script that will mirror the moodledata folder.
cd /temp/scripts
touch datafiles_backup.run
chmod +x datafiles_backup.run

5. Edit the script -  log each time the folder is mirrored. Ensure that the folder exists [/temp/logs]
nano datafiles_backup.run
Add to file:

rsync --delete-after -e ssh -avz /path/to/your/moodledata/folder/ root@192.168.100.2:/path/to/mirror
date >> /temp/logs/moodledatamirror.log Save and exit



6. Test script
cd /temp/scripts
./datafiles_backup.run
Some output should be visible and the script will notify when complete.

7. Check to see if the files went across properly!
Login via ssh to Moodle Backup server
ssh -l root 192.168.100.2
cd /path/to/mirror
ls -all

Compare the results of the list to the original folder on the Moodle Server!

8.Add the script to the crontab to execute nightly
Schedule to run this script at periods of low activity - 2:00am should be fine for most installs. Remember to factor in the Database backup time!

nano /etc/crontab
Append the following:

#database backup script
00 2 * * *  root /bin.sh /temp/scripts/datafiles_backup.run

The MoodleData files will now be mirrored nightly!
This same strategy can be used to transfer the database backups to the backup server.

-n

Restoring a Moodle Database Backup

This is an addendum to my earlier post: Backing Up your Moodle Database

Now you have a database backup. In the event that something horrible happens (knock on wood) and the database needs to be restore...
This assumes that the database is located in the default MySQL installation location: /var/lib/mysql/yourdbname

0. Login as root on the database server

1. Stop the mysqld service
service mysqld stop


2. Backup the current corrupted database (in case forensic analysis is needed later)
cp /var/lib/mysql/yourdbname /var/lib/mysql/yourdbname_backup

3. Empty the files from the database.
cd /var/lib/mysql/yourdbname
rm -rf *

4. Copy the backup to the empty database shell
cp /path/to/your/backup/* /var/lib/mysql/yourdbname/
*make sure that the folder /var/lib/mysql/yourdbname/ only contains MYI, MYD, frm and opt files ONLY. 


5.  Ensure that permissions are correctly set.
cd /var/lib/mysql/yourdbname/
chown mysql:mysql *
chmod 771 *


6. Start the Mysql Service.
service mysqld start


7. Run a Check/Repair/Optimize on the restored database.
This is to ensure consistency of the data you just restored.
There is no need to login to mysql, simply execute from the terminal.

mysqlcheck yourdbname -c -o -a -r -f -h localhost -u yourdbusername -p yourdbpassword

Note: If the database for the Moodle install is large,  a weekly or nightly optimization can be configured to minimize the chances of irrepairable corruption.
This can be done similarly to the cron job in my earlier post.

7.1 Create script
cd /temp/scripts
touch database_check.run
chmod +x database_check.run
7.2. Edit the script
nano database_check.run
Add the following (I chose to log each time the script runs)

mysqlcheck yourdbname -c -o -a -r -f -h localhost -u yourdbusername -p yourdbpassword > /temp/logs/database_check_$(date +%Y%m%d)_$(date +%H%M).txt
The log folder needs to be created so the script does not throw an error!
cd /temp
mkdir logs
7.3 Test

./temp/scripts/database_check.run
check the latest logfile for output
cat /temp/logs/database_check_xxxxxx.txt
You should see some output with the table names and status next to it - e.g
mdl_config      OK

-n