Choosing the right scheduling algorithm for a Linux Based Load Balancer

I'm currently doing some research into choosing the right scheduling algorithm for a Linux based Load balancer for a dual front end moodle installation.
The actual loadbalancer implementation will be detailed later.

This particular post will be to outline the scheduling algorithms possible on ipvs and which would be the best option for my particular scenario. The assumption in my case is that both webservers are identical, from hardware specifications to OS to php code.

From the CentOS Documentation on IPVS Scheduling Algorithms:

Round-Robin Scheduling
Distributes each request sequentially around the pool of real servers. Using this algorithm,
all the real servers are treated as equals without regard to capacity or load. This scheduling
model resembles round-robin DNS but is more granular due to the fact that it is networkconnection
based and not host-based. LVS round-robin scheduling also does not suffer the
imbalances caused by cached DNS queries.

Weighted Round-Robin Scheduling
Distributes each request sequentially around the pool of real servers but gives more jobs to
servers with greater capacity. Capacity is indicated by a user-assigned weight factor, which
is then adjusted upward or downward by dynamic load information.
Weighted round-robin scheduling is a preferred choice if there are significant differences in
the capacity of real servers in the pool. However, if the request load varies dramatically, the
more heavily weighted server may answer more than its share of requests.

Least-Connection
Distributes more requests to real servers with fewer active connections. Because it keeps
track of live connections to the real servers through the IPVS table, least-connection is a
type of dynamic scheduling algorithm, making it a better choice if there is a high degree of
variation in the request load. It is best suited for a real server pool where each member
node has roughly the same capacity. If a group of servers have different capabilities,
weighted least-connection scheduling is a better choice.

Weighted Least-Connections (default)
Distributes more requests to servers with fewer active connections relative to their capacities.
Capacity is indicated by a user-assigned weight, which is then adjusted upward or
downward by dynamic load information. The addition of weighting makes this algorithm
ideal when the real server pool contains hardware of varying capacity.

Locality-Based Least-Connection Scheduling
Distributes more requests to servers with fewer active connections relative to their destination
IPs. This algorithm is designed for use in a proxy-cache server cluster. It routes the
packets for an IP address to the server for that address unless that server is above its capacity
and has a server in its half load, in which case it assigns the IP address to the least
loaded real server.

Locality-Based Least-Connection Scheduling with Replication Scheduling
Distributes more requests to servers with fewer active connections relative to their destination
IPs. This algorithm is also designed for use in a proxy-cache server cluster. It differs
from Locality-Based Least-Connection Scheduling by mapping the target IP address to a
subset of real server nodes. Requests are then routed to the server in this subset with the
lowest number of connections. If all the nodes for the destination IP are above capacity, it
replicates a new server for that destination IP address by adding the real server with the
least connections from the overall pool of real servers to the subset of real servers for that
destination IP. The most loaded node is then dropped from the real server subset to prevent
over-replication.

Destination Hash Scheduling
Distributes requests to the pool of real servers by looking up the destination IP in a static
hash table. This algorithm is designed for use in a proxy-cache server cluster.
Source Hash Scheduling
Distributes requests to the pool of real servers by looking up the source IP in a static hash
table. This algorithm is designed for LVS routers with multiple firewalls.

I'm currently doing some testing with one or two of the more viable options and will follow up with my choice (and why I chose it).

Part II here.
-n

Using the Linux diff command

I need to do a Moodle minor version upgrade(1.9.x to 1.9.y), but my basecode is highly customized. For a vanilla installation of moodle, an upgrade is very straightforward - but how much moodle installs are actually vanilla?

What I intend to do is a side-by-side comparison to check from a file standpoint the differences between the updated moodle core and my customized version.

0. copy production moodle code to a test machine (if possible)

1. on test machine download latest version of moodle from www.moodle.org and extract to a folder in tmp

2. run the following command:
diff -qry /path/to/current/code /path/to/downloaded/code > /pipe/to/textfile.txt

3. Have fun comparing files. I recommending opening the file in a spreadsheet editor.

Cheers,
-n

Find and Delete certain files in CentOS Linux

This command can be scripted (needs to be run as root), but this script is not covered here.

Basically what this command does is search a specified directory for a name/filetype and then delete. 
If you run it from the top level directory (/), chances are something important may get deleted. Be sure to specify your folder, it will traverse subfolders.

My particular reason for using this is I need to clean up over 3000 courses in Moodle and remove any old course backups that were used for creating course copies by the site admin. This involves sorting through tens of thousands of folders for course backups.

Bear in mind that this action cannot be undone, so if Lecturers/Teachers manage their own Moodle course backups, this script will not work for you. As always, you should have backups of all necessary files before attempting this sort of activity.


In order to ensure that no unnecessary files get deleted, run the command without the delete section.
find /path/to/your/folder -name \*backup\*.zip

*edit: to pipe the list including the size of the files to a textfile, please see below:
find /path/to/your/folder -size +5k -name \*searchstring\*.zip -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' > /temp/report.txt

much thanks to http://www.cyberciti.biz/faq/find-large-files-linux/ for the size additionA list of files matching your search string will be displayed.






Breakdown:
find - Linux search command
/path/to/your/folder/ - self explanatory
-name - tells the command to search for specific filenames
\*backup\*.zip - search string, will return the following:

mybackup.zip
mybackup2001.zip
backup2002.zip

will not return:

backup.txt
mybackup.doc
backup1.log

 | xargs -   allows commands to be run from previous query/command
/bin/rm -f - delete command


Now for the cleanup:

find /path/to/your/folder -name \*backup\*.zip | xargs /bin/rm -f

Congrats! All your base are belong to us..err wait, I meant the files should have been deleted.
Rerun the original find command with your search string and it should no longer return any results.



Cheers,
-n

Installing AIDE (Advanced Intrusion Detection Environment) on CentOS 5.x

AIDE (Advanced Intrusion Detection environment) is a great approach to layered security on a Linux Server. This covers a basic install and configuration to run once per day with a report to your email account(or not).
It is recommended to run this on a "Clean" system - i.e one that is perhaps freshly installed and configured before deployment on the web.
Any updates to software or system on the server after this point will trigger false positives, so be sure to update the database after any such work is done.

0. Log in as root

1. Install the package
yum install aide

2. Edit the config file to be able to send to your email address
NOTE: If you do not wish to receive a daily report or you want to inspect the logs manually, skip this section ang go to step 3.


nano /etc/aide.conf
look for the following lines and comment out via # at the beginning of the line

report_url=file:@@{LOGFIR}/aide.log
report_url=stdout

 it should now read:

#report_url=file:@@{LOGFIR}/aide.log

#report_url=stdout

add the following lines immediately below the commented out section as mentioned above:

report_url=mailto:youremail@yourdomain.com
report_url=syslog:LOG_AUTH

save and exit /etc/aide.conf

3. Run AIDE to create the initial database
Steps 3 and 4 will need to be repeated each time you do a system update or modify any configuration files, so be warned. Security and convenience are mutually exclusive.
aide --init

4. Copy the database to default setting - this is the baseline database.
cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz


5.  Run the AIDE first check
aide --check
Expected output in a perfect world:

AIDE, version 0.13.1


### All files match AIDE database. Looks okay!

6. Setup a daily job (in this case to run at 11pm) to run
nano /etc/crontab
if you wish to receive the email as configured in step 2, add to end of file

00 23 * * * /usr/sbin/aide --check /bin/mail -s "$HOSTNAME - Daily AIDE integrity check" youremail@yourdomain.com

if you do not wish to receive any email modify the crontab as indicated below

00 20 * * * /usr/sbin/aide --check

the default logs can be accessed at:
/var/log/aide/aide.log
There will be a list of modified files and or filesystem.

Further information on AIDE as well as troubleshooting can be found here: http://www.cs.tut.fi/~rammer/aide/manual.html

Cheers,
-n

Installing Tomcat5 on CentOS 5.x

This covers pure installation of the Tomcat server on CentOS alongside Apache httpd, any configurations can be found on the Tomcat site: http://tomcat.apache.org/

0. Login as root (standard rules apply)
 
1. Install the prerequisite packages
yum install -y httpd-devel gcc-java tomcat*

2. Set Auto startup on boot
chkconfig tomcat5 on
service tomcat5 restart

3. Check that the server is operational
From a web browser on the machine (or on the network) type in the following:
http://localhost:8080 (http://your.network.ip.address:8080 from a network machine)
An Apache Tomat testpage should be displayed.

4. Upload a test script
The default tomcat directory is /var/lib/tomcat5/webapps/ROOT/
Create a sample jsp on the server named test.jsp and run it from a browser to ensure the server is operating normally.

cd /var/lib/tomcat5/webapps/ROOT/
nano test.jsp

Paste the following and save the file:
(taken from http://java.sun.com/developer/technicalArticles/xml/WebAppDev/)
**Replace [] with <>**

[HTML][HEAD]
[TITLE]JSP Example[/TITLE]
[/HEAD]
[BODY BGCOLOR="ffffcc"]
[CENTER]
[H2]Date and Time[/H2]
[% 
java.util.Date today = new java.util.Date();
out.println("Today's date is: "+today);
%]
[/CENTER]    
[/BODY]    
[/HTML] 

 5. Access the test script from a web browser
http://localhost:8080/test.jsp (http://your.network.ip.addr:8080/test.jsp)
You should get a yellow page showing the current system date/time.

Cheers,
-n

Lost your root password?

Just finished a fresh install of CentOS 5.5, and in my pangs of hunger, I found myself locked out of my root account, as I was unable to remember the bloody password I set 5 minutes ago.

This outlines how to quickly change the root password if locked out.

1. Boot into single user mode. Boot the machine and at the start up options, press a to append to the GRUB loader config.
The following text should be visible
ro root=LABEL=/
Delete any text after this and append the following line (be sure to include a space after the slash)
single
It should look like this
ro root=LABEL=/ single

The Single user prompt should appear after booting:
sh-2.05b#
2. Change the root password
Enter the following command to change the root password:
passwd root

You will be prompted to change and confirm the new root password

3. RebootUpon reboot, you should be able to access the root account.

Cheers,
-n

2011

2011 is here, and with it some more stuff.
Plans for the next few weeks/months.
NTP
High Availability Load Balancing
...more to come.

-n