Tuesday, March 23, 2010

In pursuit of a monitoring solution for CentOS Linux: Part VI - Zabbix Agent Installation

Continuation of "In pursuit of a monitoring solution for CentOS Linux"
Part I
Part II
Part III
Part IV
Part V 

Part VI - Zabbix Agent Install [Linux Specific]

0. Login as root on the server to be monitored.

1. Install prerequisite packages
yum install gcc

2. Copy Zabbix tarball and extract
mkdir /temp/zabbix && cd /temp/zabbix
copy the zabbix tarball into this folder (via scp from server, or it can be re-downloaded)
tar -xzvf zabbix-1.8.1.tar.gz
cd zabbix-1.8.1.tar.gz

3. Install the zabbix agent
./configure --enable-agent --prefix=/usr/local/zabbix
make install

4. Create the Zabbix User
useradd zabbix

5. Configure Zabbix Agent
echo 'zabbix_agent 10050/tcp' >> /etc/services
echo 'zabbix_trap 10051/tcp' >> /etc/services

6. Copy the sample configs for the agentd.
mkdir /etc/zabbix
cp misc/conf/zabbix_agentd.conf /etc/zabbix

7. Edit agentd config
nano /etc/zabbix/zabbix_agentd.conf
Server=Your.Zabbix.Server.IP
Host_name=your_server_name ***[Remember this name! Case Sensitive]

8. Configure Agent to Automatically start
cp misc/init.d/redhat/zabbix_agentd_ctl /etc/init.d/zabbix_agentd
nano /etc/init.d/zabbix_agentd
Just below #!/bin/sh: add these two lines, including the # hash marks.
# chkconfig: 345 95 95
# description: Zabbix Agentd
Edit the following parameters:
BASEDIR=/usr/local/zabbix
ZABBIX_AGENTD=$BASEDIR/sbin/zabbix_agentd
chkconfig --level 345 zabbix_agentd on 


9. Edit hostsfile to allow communication
nano /etc/hosts.allow
ALL: your.zabbix.server.ip : allow

10.Depending on your firewall setup, open the follwoing ports: 10050 and 10051.

11. Start the Zabbix Agent Service
service zabbix_agentd start

12. Make sure the agent is operational
cat /tmp/zabbix_agentd.log

The expected output is a few lines, one of which should say Zabbix Agent is running...


Now login to the Zabbix Server web interface/ GUI.


13.  Setup the host on the server
Go to Configuration –> Hosts –> Create Host
Host: Your Server Name(use the same name from step 7)
DNS name: your dns info [your dns name only]
IP address: 192.168.x.x [Your Host ip address]
Port: 10050
Status: Monitored [We will change this after installing the agent]
Link with Template: Template_Linux


Save!

Documentation of Triggers and other necessary information on further configuration can be found on the Zabbix Website.


-n

Wednesday, March 17, 2010

MySQL Maintenance for MyISAM Tables

Moodle.org recommended performing regular re-indexing of the tables in a Large Moodle Installation. Since my install of Moodle uses the MyISAM storage engine, this will be accomplished using the myisamcheck command.

This will mean stopping the MySQL service whilst the command is running!

0. Login as root on the database server.

1. Stop the Mysql Service
service mysqld stop

2. Run the myisamcheck command
myisamchk --analyze --check --extend-check --force --verbose /var/lib/mysql/yourdatabasename/ *.MYI

3. Start the Mysql Service
 service mysqld start

It is recommended to do this monthly for very large databases with high activity.

-n

Monday, March 15, 2010

In pursuit of a monitoring solution for CentOS Linux: Part V - Zabbix Server Installation

Continuation of "In pursuit of a monitoring solution for CentOS Linux"
Part I
Part II
Part III
Part IV

Part V - Zabbix Server Installation

This install was done mostly from this guide located at muck.net


0. Login as Root on the Monitoring Server (assuming the same server in the Nagios Server Setup)


1. Install prerequisite packages
yum -y install ntp php php-bcmath php-gd php-mysql httpd mysql gcc mysql-server mysql-devel net-snmp net-snmp-utils net-snmp-devel net-snmp-libs curl-devel mak


2. Fping is not part of the base repository, so the RPM's is downloaded from Dag Rep.
wget http://dag.wieers.com/rpm/packages/fping/fping-2.4-1.b2.2.el5.rf.i386.rpm
rpm -Uvh fping-2.4-1.b2.2.el5.rf.i386.rpm
chmod 7555 /usr/sbin/fping


3. Create the Zabbix User
useradd zabbix

4. Download and Install Zabbix
mkdir /temp/zabbix && cd /temp/zabbix
wget http://downloads.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Stable/1.8.1/zabbix-1.8.1.tar.gz?use_mirror=iweb
tar -xzvf zabbix-1.8.1.tar.gz

5. Start/Restart Mysql and create the empty database shell
service mysqld restart
mysql -u root -p
In MYSQL:
mysql> CREATE DATABASE zabbix character set utf8;
mysql> GRANT DROP,INDEX,CREATE,SELECT,INSERT,UPDATE,ALTER,DELETE ON zabbix.* TO zabbixmysqluser@localhost IDENTIFIED BY ‘zabbixmysqlpassword’;
mysql> quit;

6. Create the DB Schema
cd zabbix-1.8.1
cat create/schema/mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix
cat create/data/data.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix
cat create/data/images_mysql.sql | mysql -u zabbixmysqluser -pzabbixmysqlpassword zabbix
./configure --enable-server --prefix=/usr/local/zabbix –-ith-mysql --with-net-snmp --with-libcurl
make install
make clean

7. Compile the Zabbix Agent
./configure --enable-agent --prefix=/usr/local/zabbix --enable-static
make install

8. Add Zabbix Server and Agent ports to the Services file
echo ‘zabbix_agent 10050/tcp’ >> /etc/services
echo ‘zabbix_trap 10051/tcp’ >> /etc/services

9. Copy the sample configs to /etc/zabbix for server and agentd.
mkdir /etc/zabbix
cp misc/conf/zabbix_agentd.conf /etc/zabbix
cp misc/conf/zabbix_server.conf /etc/zabbix


10. Modify Zabbix server configuration
nano /etc/zabbix/zabbix_server.conf
Change the following entries
DBUser=zabbixmysqluser
BPassword=zabbixmysqlpassword
DBSocket=/var/lib/mysql/mysql.sock
FpingLocation=/usr/sbin/fping

11. Modify the Agent configuration
nano /etc/zabbix/zabbix_agentd.conf
Change the following entries
Server=127.0.0.1,Your.Zabbix.Server.IP
Hostname=UniqueHostnameForEachAgent


12. Copy Server and Agent files
cp misc/init.d/redhat/zabbix_agentd_ctl /etc/init.d/zabbix_agentd
cp misc/init.d/redhat/zabbix_server_ctl /etc/init.d/zabbix_server
Modify /etc/init.d/zabbix_agentd AND /etc/init.d/zabbix_server:
BASEDIR=/usr/local/zabbix

13. Edit Agentd and Server config
nano /etc/init.d/zabbix_agentd (Note the # hash marks, they are necessary), add near the top, just below #!/bin/sh:
# chkconfig: 345 95 95
# description: Zabbix Agentd

nano /etc/init.d/zabbix_server (again, note the # Hash marks, they are required), add near the top, just below #!/bin/sh:
# chkconfig: 345 95 95
# description: Zabbix Server


14. Configure Services to start automatically on boot.
chkconfig zabbix_server on
chkconfig zabbix_agentd on
chkconfig httpd on
chkconfig mysqld on

15. Depending on your firewall setup, open the follwoing ports:for port 80, 10050, and 10051.

16. Copy phpfiles to site root
cp -r frontends/php /var/www/html/zabbix

17. Tweak php.ini
nano /etc/php.ini
max_execution_time = 300
date.timezone = Pick a Timezone from here

18. Start Apache and make the zabbix folder writable
service httpd start
chmod 777 /var/www/html/zabbix/conf

19. Launch http://your.server.ip/zabbix in your web browser.
A Setup Screen should be displayed.
Click the EULA, and confirm all prerequisites are met on the next screen.
Setup the DB connection using the user and password set in step 6.

20. When the setup is complete, the webroot needs to be secured and the services restarted.
chmod 755 /var/www/html/zabbix/conf
mv /var/www/html/zabbix/setup.php /var/www/html/zabbix/setup.php.bak
service zabbix_agentd start
service zabbix_server start

21. Zabbix is now accessible using http://your.server.ip/zabbix
username: admin
Password:
(no password)


On to configuring the Zabbix Agents!

-n

Tuesday, March 9, 2010

In pursuit of a monitoring solution for CentOS Linux: Part IV - Nagios Client Installation

Continuation of "In pursuit of a monitoring solution for CentOS Linux"
Part I
Part II
Part III
Part IV - Nagios Client Installation

This is accomplished using Nagios Plugins and the NRPE Daemon to report information to the Monitoring Server.

0. Login as root on Client (Server to be monitored)

1. Install prerequisite packages - openssl-devel and xinetd is needed.
yum install openssl-devel xinetd

2. Create nagios account
useradd nagios
passwd nagios


3. Create folder to store downloaded files
mkdir -p /opt/Nagios/Nagios_Plugins && cd /opt/Nagios/Nagios_Plugins
**Go to http://www.nagios.org/download/download.php and get the URL for the latest versions of the software. Use wget to download to directory.

4. Extract Files
tar xzf nagios-plugins-1.4.13.tar.gz
cd nagios-plugins-1.4.13


5. Compile and Configure plugins
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
make install


6. Change permissions on plugins and plugin directory
chown nagios.nagios /usr/local/nagios
chown -R nagios.nagios /usr/local/nagios/libexec


Install NRPE Daemon

7. Create folder to store downloaded files
mkdir -p /opt/Nagios/Nagios_NRPE && cd /opt/Nagios/Nagios_NRPE
**Go to http://www.nagios.org/download/download.php and get the URL for the latest versions of the software. Use wget to download to directory.

8. Extract the files
tar -xzf nrpe-2.12.tar.gz
cd nrpe-2.12


9. Compile and Configure NRPE

./configure

Exected Output
General Options:
-------------------------
NRPE port: 5666
NRPE user: nagios
NRPE group: nagios
Nagios user: nagios
Nagios group: nagios

make all
make install-plugin
make install-daemon
make install-daemon-config
make install-xinetd


10. NRPE Configutation
nano /etc/xinetd.d/nrpe
edit line:
only_from = your.nagios.SERVER.address

nano /etc/services

add line:
nrpe 5666/tcp # NRPE

11. Restart Xinetd and Set to start at boot
chkconfig xinetd on
service xinetd restart


12. Open Port 5666 on Firewall
system-config-securitylevel-tui

13. Test NRPE Daemon

netstat -at |grep nrpe
Expected Output:
tcp 0 0 *:nrpe *.* LISTEN

/usr/local/nagios/libexec/check_nrpe -H localhost
Expected Output:
NRPE v2.12


At this point further configuration is needed on the Nagios Server, to accept data from the client.

14. Login as root on Nagios Server

15. Downlad and Install NRPE Plugin

mkdir -p /opt/Nagios/Nagios_NRPE && cd /opt/Nagios/Nagios_NRPE
**Go to http://www.nagios.org/download/download.php and get the URL for the latest versions of the software. Use wget to download to directory.

16. Extract the Files
tar -xzf nrpe-2.12.tar.gz
cd nrpe-2.12


17. Compile and Configure NRPE
./configure
make all
make install-plugin


18. Test connection to client
/usr/local/nagios/libexec/check_nrpe -H client.ip.address
Expected Output:
NRPE v2.12


19.Create NRPE Command Definition

nano /usr/local/nagios/etc/objects/commands.cfg

Add the following:
###############################################################################
# NRPE CHECK COMMAND
#
# Command to use NRPE to check remote host systems
###############################################################################

define command{
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}


20. Create Linux Object template
nano /usr/local/nagios/etc/objects/linux-box-remote.cfg

Add the following and replace the values “host_name” “alias” “address” with the values that match your setup:
** The “host_name” you set for the “define_host” section must match the “host_name” in the “define_service” section **
define host{
name linux-box-remote ; Name of this template
use generic-host ; Inherit default values
check_period 24x7
check_interval 5
retry_interval 1
max_check_attempts 10
check_command check-host-alive
notification_period 24x7
notification_interval 30
notification_options d,r
contact_groups admins
register 0 ; DONT REGISTER THIS - ITS A TEMPLATE
}

define host{
use linux-box-remote ; Inherit default values from a template
host_name Centos5 ; The name we're giving to this server
alias Centos5 ; A longer name for the server
address 192.168.0.5 ; IP address of the server
}

define service{
use generic-service
host_name Centos5
service_description CPU Load
check_command check_nrpe!check_load
}
define service{
use generic-service
host_name Centos5
service_description Current Users
check_command check_nrpe!check_users
}
define service{
use generic-service
host_name Centos5
service_description /dev/hda1 Free Space
check_command check_nrpe!check_hda1
}
define service{
use generic-service
host_name Centos5
service_description Total Processes
check_command check_nrpe!check_total_procs
}
define service{
use generic-service
host_name Centos5
service_description Zombie Processes
check_command check_nrpe!check_zombie_procs
}

21. Activate the linux-box-remote.cfg template
nano /usr/local/nagios/etc/nagios.cfg
And add:
# Definitions for monitoring remote Linux machine
cfg_file=/usr/local/nagios/etc/objects/linux-box-remote.cfg

22. Verify Nagios Config Files

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
Expected Output:
Total Warnings: 0
Total Errors: 0

23. Restart Nagios Service
service nagios restart


Please check the Nagios website if any errors are encountered. This install went off relatively smoothly, no major errors.

Only the standatd NRPE plugins were demonstrated in this install, please visit the Nagios website for more plugins!

-n

In pursuit of a monitoring solution for CentOS Linux: Part III - Nagios Server Installation

Continuation of "In pursuit of a monitoring solution for CentOS Linux"
Part I
Part II


Part III and IV covers installation of the Nagios Server and configuration of the Clients.

Nagios Server Install on CentOS 5.x

For the purpose of this install, Server will refer to the monitoring server and Client will refer to the servers being monitored.

0. Log in to Server as root

1. Install the necessary pacakges

yum install httpd gcc glibc glibc-common gd gd-devel php

2. Create user and group for Nagios
useradd -m nagios
groupadd nagcmd
usermod -a -G nagcmd nagios
usermod -a -G nagcmd apache


3. Create directory to store Nagios Downloaded files

mkdir /opt/Nagios && cd /opt/Nagios
**Go to http://www.nagios.org/download/download.php and get the URL for the latest versions of the software. Use wget to download to directory.

4. Extract files
cd /opt/Nagios
tar xzf nagios-3.0.6.tar.gz
cd nagios-3.0.6


5. Compile and Configure
./configure --with-command-group=nagcmd
make all
make install
make install-init
make install-config
make install-commandmode
make install-webconf


6. Create Web Interface Login User
Create user “nagiosadmin” ( remember the password assigned !)
htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

7. Restart Apache
service httpd restart


Install the Nagios Plugins

8. Extract files
cd /opt/Nagios
tar xzf nagios-plugins-1.4.13.tar.gz
cd nagios-plugins1.4.13


9. Compile and Configure Nagios Plugins
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
make install


10. Configure the email address to send notifications to
nano /usr/local/nagios/etc/objects/contacts.cfg
email nagios@localhost ; << CHANGE THIS TO YOUR EMAIL ADDRESS

11. Verify config file
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Expected output
Total Warnings: 0
Total Errors: 0

12. Start Nagios and set to start automatically on reboot.
chkconfig --add nagios
chkconfig nagios on
chkconfig httpd on
service nagios start


13. Log on to server from an internet browser
http://server.ip.address/nagios/

On to Part IV - Nagios Client install/configuration

-n

Monday, March 8, 2010

In pursuit of a monitoring solution for CentOS Linux: Part II - The Winners

This is an update to my earlier post: In pursuit of a monitoring solution for CentOS Linux

Based on some testing for the past couple weeks, the winner, or rather winners turned out to be Nagios and Zabbix.

Nagios was a great monitoring solution and was highly customizable and the plugins were simple to modify.

Zabbix is a lot more complex, but offers very nifty graphs for analysis. Graphs are always good ;)

Future posts will have the Nagios and Zabbix installation instructions for CentOS5.x - both the Client and the Server.

-n

Monday, March 1, 2010

Installing memcached on CentOS 5.x

Memcached is a distributed memory object caching system - as quoted from http://memcached.org/

In other words, it is simply a cache for common database queries and typically reduces database load by caching the mentioned queries.

It is not part of the standard CentOS distro, so installation of the DAG repositories was necessary.

0. Login as root on your webserver(s)

This install specific to the 32-bit version of CentOS 5.x

1. Get the rpmforge rpm (if not already installed)




2. Install the rpmforge rpm
rpm -Uvh rpmforge-release-0.3.6-1.el5.rf.i386.rpm

3. Install memcached and other packages required
yum install --enablerepo=rpmforge memcached php-pecl-memcache


4. Modify php.ini
nano /etc/php.ini

add line under Dynamic Extension section.
;  extension=memcache.so



5. Create memcached configuration file
nano /etc/sysconfig/memcached
Add to file:
#  Specify Port
PORT="11211"
#Specify User for service to run.
USER="memcached"
#Specify maximum number of connections
MAXCONN="2048"
#Set Cache size based on Memory available

CACHESIZE="512"
#Specify which interface to listen to - Security Measure
OPTIONS="-l 127.0.0.1"

6. Add the memcached user (the same username entered in step 5)
useradd memcached

7. Start the memcached daemon and set to automatically start on reboot
service memcached start
chkconfig memcached on


8. Test memcached
Original Script located at http://dorkage.net/2009/02/memcached-test/
Create script in site root, or other preferred location.
touch /var/www/html/memcachedtest.php

Paste the following code:

<?php
//memcached simple test
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("Could not connect");
$key = md5('42data');  //something unique
for ($k=0; $k<5; $k++) {
$data = $memcache->get($key);
    if ($data == NULL) {
        $data = array();
        //generate an array of random shit
        echo "expensive query";
        for ($i=0; $i<100; $i++) {
            for ($j=0; $j<10; $j++) {
                $data[$i][$j] = 42;  //who cares
            }
        }
        $memcache->set($key,$data,0,3600);
    } else {
        echo "cached";
    }
}
?>



Run http://yourserver.com/memcachedtest.php to see if it works
You should see "Expensive Query"
If memcached works, you should see "Cached Cached Cached" upon refreshing the page.

-n