I was running into some issues with the EPEL repo on CentOS 6.x. I had just installed the OS and the EPEL/remi repos and whilst trying to search for a package the same error kept repeating.
Cannot retrieve metalink for repository: epel
and it kept failing each time.
The fix involved editing the EPEL repo and making a simple change from https to http. Why or how this broke is beyond me, but this fix may help someone else.
Assuming you have root or su:
1. Edit the epel repo
nano /etc/yum.repos.d/epel.repo
Find line:
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
Change to
mirrorlist=http://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
Save and exit.
2. Clean yum
yum clean all
That did it for me!
Cheers,
noveck
Content moved to tucuche-consulting.com as of April 2019
Wednesday, January 16, 2013
Wednesday, January 9, 2013
InnoDB Backup Scripts
Following up on my earlier post about converting the storage engine on a MySQL Database from MyISAM to InnoDB, I'd like to share the following scripts with backup rotation built in.
The next blog entry should be a step-by-step restore procedure using the actual backups below.
Test Environment:
CentOS 5.x
MySQL 5.0 (distro version)
Percona Xtrabackup 2.0.3 installed as per this howto
Backup Directory Structure
/bkp
/bkp/Hourly
/bkp/Daily
/bkp/Monthly
Script Output Log Directory
/tmp
Script Directory
/scripts
Disclaimer: I do not guarantee this is the BEST way of doing this, but it works for me. Copy and paste the following into a *.sh file in your scripts directory. Ensure the executable flag is set
chmod +x scriptname.sh
Actual Scripts
innodb_backup_monthly.sh
#!/bin/sh
# An InnoDB Backup Script to backup database Monthly
#
# Written by: Noveck Gowandan
# 02-10-2012
# Version 1.1
# Modified filename convention
# Uses tar-gzip to further compress final archive
# Added script timer and modified output to log time
# Start timer
time_start=`date +%s`
# Go to backup location and create Monthly folder with datestamp
cd /bkp/
mkdir M_db_$(date +%Y%m%d)
# Execute backup using innobackupex and send to folder created previously
innobackupex --defaults-file=/etc/my.cnf --user=****** --password=****** --databases=mydbname /bkp/M_db_$(date +%Y%m%d)
# Compress backup into a tarball
tar czf mydbname_$(date +%Y%m%d).tgz M_db*
# Backup rotation section
rm -rf M_db*
rm -rf /bkp/Monthly/$(date +"%B")
mkdir /bkp/Monthly/$(date +"%B")
mv mydbname* /bkp/Monthly/$(date +"%B")
# Stop timer and calculate total time
time_end=`date +%s`
total_time=`expr $(( $time_end - $time_start ))`
# Log output: datestamp and time takes to execute
echo "____________" >> /tmp/db_backup_monthly.log
date >> /tmp/db_backup_monthly.log
echo "Execution Time was $total_time seconds." >> /tmp/db_backup_monthly.log
# Go to backup location and create Daily folder with datestamp
cd /bkp/
mkdir D_db_$(date +%Y%m%d)
# Execute backup using innobackupex and send to folder created previously
innobackupex --defaults-file=/etc/my.cnf --user=****** --password=****** --databases=mydbname /bkp/D_db_$(date +%Y%m%d)
# Compress backup into a tarball
tar czf mydbname_$(date +%Y%m%d).tgz D_db*
# Backup rotation section
rm -rf D_db*
rm -rf /bkp/Daily/$(date +"%A")
mkdir /bkp/Daily/$(date +"%A")
mv mydbname* /bkp/Daily/$(date +"%A")
# Stop timer and calculate total time
time_end=`date +%s`
total_time=`expr $(( $time_end - $time_start ))`
# Log output: datestamp and time takes to execute
echo "____________" >> /tmp/db_backup_daily.log
date >> /tmp/db_backup_daily.log
echo "Execution Time was $total_time seconds." >> /tmp/db_backup_daily.log
innodb_backup_hourly.sh
The next blog entry should be a step-by-step restore procedure using the actual backups below.
Test Environment:
CentOS 5.x
MySQL 5.0 (distro version)
Percona Xtrabackup 2.0.3 installed as per this howto
Backup Directory Structure
/bkp
/bkp/Hourly
/bkp/Daily
/bkp/Monthly
Script Output Log Directory
/tmp
Script Directory
/scripts
Disclaimer: I do not guarantee this is the BEST way of doing this, but it works for me. Copy and paste the following into a *.sh file in your scripts directory. Ensure the executable flag is set
chmod +x scriptname.sh
Actual Scripts
innodb_backup_monthly.sh
#!/bin/sh
# An InnoDB Backup Script to backup database Monthly
#
# Written by: Noveck Gowandan
# 02-10-2012
# Version 1.1
# Modified filename convention
# Uses tar-gzip to further compress final archive
# Added script timer and modified output to log time
# Start timer
time_start=`date +%s`
# Go to backup location and create Monthly folder with datestamp
cd /bkp/
mkdir M_db_$(date +%Y%m%d)
# Execute backup using innobackupex and send to folder created previously
innobackupex --defaults-file=/etc/my.cnf --user=****** --password=****** --databases=mydbname /bkp/M_db_$(date +%Y%m%d)
# Compress backup into a tarball
tar czf mydbname_$(date +%Y%m%d).tgz M_db*
# Backup rotation section
rm -rf M_db*
rm -rf /bkp/Monthly/$(date +"%B")
mkdir /bkp/Monthly/$(date +"%B")
mv mydbname* /bkp/Monthly/$(date +"%B")
# Stop timer and calculate total time
time_end=`date +%s`
total_time=`expr $(( $time_end - $time_start ))`
# Log output: datestamp and time takes to execute
echo "____________" >> /tmp/db_backup_monthly.log
date >> /tmp/db_backup_monthly.log
echo "Execution Time was $total_time seconds." >> /tmp/db_backup_monthly.log
innodb_backup_monthly.sh
#!/bin/sh
# An InnoDB Backup Script to backup database DAILY
#
# Written by: Noveck Gowandan
# 02-10-2012
# Version 1.1
# Modified filename convention
# Uses tar-gzip to further compress final archive
# Added script timer and modified output to log time
# An InnoDB Backup Script to backup database DAILY
#
# Written by: Noveck Gowandan
# 02-10-2012
# Version 1.1
# Modified filename convention
# Uses tar-gzip to further compress final archive
# Added script timer and modified output to log time
# Start timer
time_start=`date +%s`
time_start=`date +%s`
# Go to backup location and create Daily folder with datestamp
cd /bkp/
mkdir D_db_$(date +%Y%m%d)
# Execute backup using innobackupex and send to folder created previously
innobackupex --defaults-file=/etc/my.cnf --user=****** --password=****** --databases=mydbname /bkp/D_db_$(date +%Y%m%d)
# Compress backup into a tarball
tar czf mydbname_$(date +%Y%m%d).tgz D_db*
# Backup rotation section
rm -rf D_db*
rm -rf /bkp/Daily/$(date +"%A")
mkdir /bkp/Daily/$(date +"%A")
mv mydbname* /bkp/Daily/$(date +"%A")
# Stop timer and calculate total time
time_end=`date +%s`
total_time=`expr $(( $time_end - $time_start ))`
# Log output: datestamp and time takes to execute
echo "____________" >> /tmp/db_backup_daily.log
date >> /tmp/db_backup_daily.log
echo "Execution Time was $total_time seconds." >> /tmp/db_backup_daily.log
innodb_backup_hourly.sh
#!/bin/sh
# An InnoDB Backup Script to backup database HOURLY
#
# Written by: Noveck Gowandan
# 02-10-2012
# Version 1.1
# Modified filename convention
# Added script timer and modified output to log time
# An InnoDB Backup Script to backup database HOURLY
#
# Written by: Noveck Gowandan
# 02-10-2012
# Version 1.1
# Modified filename convention
# Added script timer and modified output to log time
# Start timer
time_start=`date +%s`
time_start=`date +%s`
# Go to backup location and create Hourly folder with datestamp
cd /bkp/
mkdir H_db_$(date +%Y%m%d)
cd /bkp/
mkdir H_db_$(date +%Y%m%d)
# Execute backup using innobackupex and send to folder created previously
innobackupex --defaults-file=/etc/my.cnf --user=****** --password=****** --databases=mydbname /bkp/H_db_$(date +%Y%m%d)
innobackupex --defaults-file=/etc/my.cnf --user=****** --password=****** --databases=mydbname /bkp/H_db_$(date +%Y%m%d)
# Backup rotation section
rm -rf /bkp/Hourly/$(date +"%H")
mkdir /bkp/Hourly/$(date +"%H")
mv H_db* /bkp/Hourly/$(date +"%H")
rm -rf /bkp/Hourly/$(date +"%H")
mkdir /bkp/Hourly/$(date +"%H")
mv H_db* /bkp/Hourly/$(date +"%H")
# Stop timer and calculate total time
time_end=`date +%s`
total_time=`expr $(( $time_end - $time_start ))`
time_end=`date +%s`
total_time=`expr $(( $time_end - $time_start ))`
# Log output: datestamp and time takes to execute
echo "____________" >> /tmp/db_backup_hourly.log
date >> /tmp/db_backup_hourly.log
echo "Execution Time was $total_time seconds." >> /tmp/db_backup_hourly.log
echo "____________" >> /tmp/db_backup_hourly.log
date >> /tmp/db_backup_hourly.log
echo "Execution Time was $total_time seconds." >> /tmp/db_backup_hourly.log
Labels:
CentOS,
innobackupex,
InnoDB,
Linux,
MySQL,
Percona,
xtrabackup
Wednesday, November 21, 2012
Presentation: Cloud Computing - An Introduction to the Cloud
This was a presentation I did today, a basic introduction to Cloud Computing.
Enjoy!
http://prezi.com/svz06iipozjm/cloud-computing/?auth_key=16c214823b495245cc593e16948f09b1e80818a3
-noveck
Enjoy!
http://prezi.com/svz06iipozjm/cloud-computing/?auth_key=16c214823b495245cc593e16948f09b1e80818a3
-noveck
Labels:
Presentation,
Prezi
Monday, November 5, 2012
Batch convert MyISAM tables to InnoDB from Terminal
A couple months ago I posted on a modified php script to convert tables MySQL tables from MyISAM to InnoDB. It worked all well and good for my test environment, but I realized that it may not be the best solution for a dedicated database server where php is not installed.
In this case, this post assumes you have both su access and mysql root access in order to batch generate the queries to convert the storage engine. I strongly recommend isolating the database from the front end at this point, it means shutting your application down until the operation has been complete.
Please TEST before running in your production environment!
Credit to this post on RackerHacker for the kickstart query generation.
This post does not cover the advantages/disadvantages of either storage engine, that is covered at MySQL's site and at the Oracle Blog.
The benefits of Moodle on InnoDB is covered here.
Some prerequisites to get out of the way.
a. Ensure that InnoDB is enabled on the server
mysql -u root -p *****
SHOW ENGINES;
Expected outputb. Disable the writeback cache (as per MySQL's recommendation)
...
InnoDB | YES | Supports transactions...
hdparm -W0 /dev/sdx (where sdx contains the /var/lib/mysql or your specific mysql location )
c. Tune the my.cnf according to MySQL's recommendation
Later on, the Day32's Tuning Primer
and the Mysql Tuner Script can be used to fine tune the other server parameters.Prereq's aside, let's push forward onto the crux of the matter.
0. Login as root on your database server.
1. Take a backup before proceeding any further.
2. Double check that you have a backup. Please.
3. Run a check on the database to avoid a GIGO issue.
mysqlcheck -c -o -r -f -h localhost -u mydbusername -p mydbname
<enter your db password>
4. Login as MySQL root and generate the batch queries, send to output file (note the single quotes plus backticks)
select concat(‘ALTER TABLE `’,table_schema,’`.`’,table_name,’` ENGINE=InnoDB;’)
from information_schema.tables
Where table_schema=’mydbname’
and ENGINE=’MyISAM’
into outfile ‘/tmp/InnoBatchConvert.sql’
5. Logout from MySQL root and back to shell
At the shell:
mysql -u root -p --verbose < /tmp/InnoBatchConvert.sql
<enter password>
6. After the script has completed, check all the tables to ensure the storage engine is InnoDB
select table_name, engine
from information_schema.tables
where table_schema = 'mydatabasename';
7. Check the database again.
mysqlcheck -c -h localhost -u mydbusername -p mydbname
<enter your db password>
That's it! All you need to do now is ensure you have a decent backup strategy in place and the InnoDB backup script is covered here. I'll follow up soon with a detailed restore procedure as well as perhaps my own version of my innodb backup scripts with rotation built in.
-noveck
Labels:
Convert Database,
InnoDB
Tuesday, October 2, 2012
Send email from command line through an MS Exchange Relay (Alt title: Installing SSMTP on CentOS 5.x)
The situation required me to use an existing MS Exchange Server as a mail relay to send an email from a custom notification shell script I'm working on.
I tried configuring Postfix, Sendmail, Mailx, Nail, all to no avail. This took quite some time in troubleshooting and Google was not particularly helpful.
Eventually I stumbled across this post, which mentioned using the EPEL repo's to install SSMTP. Simple, but effective.
Therefore, this entry focuses on how to setup your server to relay mail, using the SSMTP package (because it works)
0. Login as root or su.
1. Check to ensure that the permissions are correct to relay mail.
telnet my.emailserver.ip.address 25
EHLO
MAIL FROM: myvalidsenderaddress@foobar.org
RCPT TO: myvalidrecipient@foobar.org
DATA
This is a test message.
. <Hit Enter>
quit
You should have received an email if the permissions are correct.
2. Install the EPEL repo
cd /tmp
wget http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
Please use this alternative if the original does not work. - http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm
3. Remove sendmail
yum remove sendmail
4. Install SSMTP
yum install ssmtp --enablerepo=EPEL
5. Configuration
Backup original conf file
cp /etc/ssmtp/ssmtp.conf /etc/ssmtp/ssmtp.conf.bkp
Edit the conf file
nano /etc/ssmtp/ssmtp.conf
Edit the following lines:
*Depending on your configuration, you may or may not need TLS, check the SSMTP.conf man for details.
6. Restart service
service ssmtp restart
7. Test the application
echo "This is the body of my email" | ssmtp myemail@foobar.org
That's it!
This worked for me where all the others mysteriously failed. Now I'm off to finish my script. I'll post another update soon!
-noveck
I tried configuring Postfix, Sendmail, Mailx, Nail, all to no avail. This took quite some time in troubleshooting and Google was not particularly helpful.
Eventually I stumbled across this post, which mentioned using the EPEL repo's to install SSMTP. Simple, but effective.
Therefore, this entry focuses on how to setup your server to relay mail, using the SSMTP package (because it works)
0. Login as root or su.
1. Check to ensure that the permissions are correct to relay mail.
telnet my.emailserver.ip.address 25
EHLO
MAIL FROM: myvalidsenderaddress@foobar.org
RCPT TO: myvalidrecipient@foobar.org
DATA
This is a test message.
. <Hit Enter>
quit
You should have received an email if the permissions are correct.
2. Install the EPEL repo
cd /tmp
wget http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
Please use this alternative if the original does not work. - http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm
3. Remove sendmail
yum remove sendmail
4. Install SSMTP
yum install ssmtp --enablerepo=EPEL
5. Configuration
Backup original conf file
cp /etc/ssmtp/ssmtp.conf /etc/ssmtp/ssmtp.conf.bkp
Edit the conf file
nano /etc/ssmtp/ssmtp.conf
Edit the following lines:
mailhub=my.emailserver.ip.address:25
Hostname=ValidDomainHostName.foobar.org
*Depending on your configuration, you may or may not need TLS, check the SSMTP.conf man for details.
6. Restart service
service ssmtp restart
7. Test the application
echo "This is the body of my email" | ssmtp myemail@foobar.org
That's it!
This worked for me where all the others mysteriously failed. Now I'm off to finish my script. I'll post another update soon!
-noveck
Monday, September 17, 2012
Installation of VNC Server on CentOS 5.x
I honestly thought I documented this already, but I searched the blog and was unable to locate it.
This covers installation and quick configuration of VNC on a server, in the event that Desktop access is needed on a remote server. In some cases, the CLI is not enough, and I've found this very useful.
On to the entry:
1. Login as root (or su)
2. Install required packages:
yum -y install vnc-server
3. Secure the VNC server:
vncpasswd
Set mypassword
cd ~/.vnc
ls
the passwd folder should be listed
4. Configure the server:
nano /etc/sysconfig/vncservers
add to end of file:
5. Start the VNC service and configure service to autostart on boot:
service vncserver start
chkconfig vncserver on
6. Further configuration:
cd ~/.vnc
nano xstartup
add under the line "# Add the foll..."
Service vncserver restart
8. Configure firewall to allow VNC server traffic
system-config-securitylevel-tui
add 5903:tcp to exceptions list
9. Open VNC client from remote peer and connect to VNC Server:
server: 192.168.x.x:3
username myusername
password: mypassword (as set in Step 3)
finito
-noveck
This covers installation and quick configuration of VNC on a server, in the event that Desktop access is needed on a remote server. In some cases, the CLI is not enough, and I've found this very useful.
On to the entry:
1. Login as root (or su)
2. Install required packages:
yum -y install vnc-server
3. Secure the VNC server:
vncpasswd
Set mypassword
cd ~/.vnc
ls
the passwd folder should be listed
4. Configure the server:
nano /etc/sysconfig/vncservers
add to end of file:
VNCSERVERS="3:myusername"
VNCSERVERARGS [3] = "-geometry 1024x768"
5. Start the VNC service and configure service to autostart on boot:
service vncserver start
chkconfig vncserver on
6. Further configuration:
cd ~/.vnc
nano xstartup
add under the line "# Add the foll..."
(while true ; do xterm ; done) &remove comments(#) from
#unset SESSION_MANAGER7. Restart the vnc service
#exec /etc/x11/xinit/xinitrc
Service vncserver restart
8. Configure firewall to allow VNC server traffic
system-config-securitylevel-tui
add 5903:tcp to exceptions list
9. Open VNC client from remote peer and connect to VNC Server:
server: 192.168.x.x:3
username myusername
password: mypassword (as set in Step 3)
finito
-noveck
Labels:
CentOS,
VNC,
vnc-client,
vnc-server
Sunday, September 9, 2012
LinuxCon2012!
Just got back from Linux Foundation's LinuxCon2012 (Aug 29-31) in San Diego.
Some great sessions and the slides were made public. It was a powerful reminder of the immense possibilities of open source and who are some of the companies running Linux in their enterprise.
See Linux Foundation's site for some of the brilliant presentations. One of my personal favourites was the fact that SpaceX's shuttle is running on linux.
Lots of great ideas in hand, stay tuned for some fresh projects!
-noveck
Some great sessions and the slides were made public. It was a powerful reminder of the immense possibilities of open source and who are some of the companies running Linux in their enterprise.
See Linux Foundation's site for some of the brilliant presentations. One of my personal favourites was the fact that SpaceX's shuttle is running on linux.
Lots of great ideas in hand, stay tuned for some fresh projects!
-noveck
Labels:
2012,
Conference,
Linux,
LinuxCon
Subscribe to:
Posts (Atom)