Wednesday, September 3, 2014

Script to monitor any service running or not

There are cases where we want to monitor running services in our servers and if in case they fails take a action according to that..

In the example script we are monitoring PHP, so if your monitoring any other service please adjust your at least "count"parameter accordingly..

Place the script in cron to check the required service..

#!/bin/bash

#To count the no of process running
count=$(ps -ef | grep -c php)
#To show count no for testing
echo $count

#Comparing the count variable with min required by us
if [ $count -lt 5 ]; then
echo -e "\nThe php seems to be not running starting it..."

#Auto starting the required service if not running, will differ according to your need
killall php-fpm
/etc/init.d/php-fpm start
/etc/init.d/nginx restart


#Sending a mail to desired user about the start of service with fetching hostname from AWS
echo -e "The PHP was stopped somehow but now it has been restarted in `wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`..\nThe PHP service is running fine now..\n\n `ps -ef | grep php`\n\n If in case PHP output is empty above please run following commands via log in to server..\n\nkillall php-fpm\n/etc/init.d/php-fpm start \n/etc/init.d/nginx restart" | mailx -r "phpalert@yourdomain.com" -s "PHP was again started in following server `date`" "ravi@theteamie.com"

 else
    echo "PHP is running fine is up and running"
  fi

The output of script will be as follows :

The PHP was stopped somehow but now it has been restarted in ec2-54-XX.XX.XX.ap-southeast-1.compute.amazonaws.com..
The PHP service is running fine now..

 root      1859  1856  0 11:50 ?        00:00:00 /bin/bash /usr/scripts/php_check.sh
root      1879     1  1 11:50 ?        00:00:00 php-fpm: master process (/etc/php-fpm.conf)
root      1882  1879 29 11:50 ?        00:00:00 /usr/lib/appdynamics-php5/proxy/jre/bin/java -server -Xmx120m -classpath /usr/lib/appdynamics-php5/proxy/conf/logging/*:/usr/lib/appdynamics-php5/proxy/lib/*:/usr/lib/appdynamics-php5/proxy/lib/tp/*:/usr/lib/appdynamics-php5/proxy/* -Djava.library.path=/usr/lib/appdynamics-php5/proxy/lib/tp -Dappdynamics.agent.logs.dir=/usr/lib/appdynamics-php5/logs -Dcomm=/tmp/ad-bmjfkD -DagentType=PHP_APP_AGENT -Dappdynamics.agent.runtime.dir=/usr/lib/appdynamics-php5/proxy com.appdynamics.ee.agent.proxy.kernel.Proxy
apache    1883  1879  0 11:50 ?        00:00:00 php-fpm: pool www0
apache    1884  1879  0 11:50 ?        00:00:00 php-fpm: pool www0
apache    1885  1879  0 11:50 ?        00:00:00 php-fpm: pool www0
root      1911  1859  0 11:50 ?        00:00:00 /bin/bash /usr/scripts/php_check.sh
root      1912  1859  0 11:50 ?        00:00:00 /bin/bash /usr/scripts/php_check.sh
root      1916  1911  0 11:50 ?        00:00:00 /bin/bash /usr/scripts/php_check.sh
root      1918  1916  0 11:50 ?        00:00:00 grep php

 If in case PHP output is empty above please run following commands via log in to server..

killall php-fpm
/etc/init.d/php-fpm start
/etc/init.d/nginx restart




Friday, August 22, 2014

How to take scheduled back up of AWS RDS snapshots

Amazon provides us with option of Automated backup but if in case you need to take scheduled manual back up than following script can be used..

In the script we are considering the scenario of taking monthly backup on 1st of every month from the daily automated RDS snapshots..

#!/bin/sh

# The script to take monthly backup of DB snapshots
# By Ravi Gadgil

#Following command will find the snapshot identifier from the DB that need to be manually copied..
aws rds describe-db-snapshots --db-instance-identifier DB-server-name1 | grep `date +%Y-%m-01` | grep rds | awk '{ print $8 }' > /tmp/DB-server-name1.txt

aws rds describe-db-snapshots --db-instance-identifier DB-server-name2 | grep `date +%Y-%m-01` | grep rds | awk '{ print $8 }' > /tmp/DB-server-name2.txt

aws rds describe-db-snapshots --db-instance-identifier DB-server-name3 | grep `date +%Y-%m-01` | grep rds | awk '{ print $8 }' > /tmp/DB-server-name3.txt


#Following command will take the snapshot identifier from above command and copy it..
aws rds copy-db-snapshot --source-db-snapshot-identifier `cat /tmp/DB-server-name1.txt` --target-db-snapshot-identifier DB-server-name1-monthly-backup-`date +%Y-%m`

aws rds copy-db-snapshot --source-db-snapshot-identifier `cat /tmp/DB-server-name2.txt` --target-db-snapshot-identifier DB-server-name2-monthly-backup-`date +%Y-%m`
aws rds copy-db-snapshot --source-db-snapshot-identifier `cat /tmp/DB-server-name3.txt` --target-db-snapshot-identifier DB-server-name3-monthly-backup-`date +%Y-%m`



We are running the above script on the 2nd of every month so it can take DB snapshot of one day old and backup it..

We have define the DB names individully in commands but you can use for loop as well if list is very big in your case..

Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.

Wednesday, April 23, 2014

Script to create daily AMI of AWS server and delete AMI older than n no of days..

Following Script can be used to create AMI of AWS server daily to keep the backup and delete the AMI older than n no of days including snapshots associated with it..

#!/bin/bash
#Script to create AMI of server on daily basis and deleting AMI older than n no of days
#By Ravi Gadgil

echo -e "----------------------------------\n   `date`   \n----------------------------------"

#To create a unique AMI name for this script
echo "instance-`date +%d%b%y`" > /tmp/aminame.txt

echo -e "Starting the Daily AMI creation: `cat /tmp/aminame.txt`\n"

#To create AMI of defined instance
aws ec2 create-image --instance-id i-1xxxxx34 --name "`cat /tmp/aminame.txt`" --description "This is for Daily auto AMI creation" --no-reboot | grep -i ami | awk '{print $4}' > /tmp/amiID.txt

#Showing the AMI name created by AWS
echo -e "AMI ID is: `cat /tmp/amiID.txt`\n"

echo -e "Looking for AMI older than 3 days:\n "

#Finding AMI older than 3 days which needed to be removed
echo "instance-`date +%d%b%y --date '4 days ago'`" > /tmp/amidel.txt

#Finding Image ID of instance which needed to be Deregistered
aws ec2 describe-images --filters "Name=name,Values=`cat /tmp/amidel.txt`" | grep -i imageid | awk '{ print  $4 }' > /tmp/imageid.txt

if [[ -s /tmp/imageid.txt ]];
then

echo -e "Following AMI is found : `cat /tmp/imageid.txt`\n"

#Find the snapshots attached to the Image need to be Deregister
aws ec2 describe-images --image-ids `cat /tmp/imageid.txt` | grep snap | awk ' { print $4 }' > /tmp/snap.txt

echo -e "Following are the snapshots associated with it : `cat /tmp/snap.txt`:\n "
 
echo -e "Starting the Deregister of AMI... \n"

#Deregistering the AMI 
aws ec2 deregister-image --image-id `cat /tmp/imageid.txt`

echo -e "\nDeleting the associated snapshots.... \n"

#Deleting snapshots attached to AMI
for i in `cat /tmp/snap.txt`;do aws ec2 delete-snapshot --snapshot-id $i ; done

else

echo -e "No AMI found older than minimum required no of days"
fi

 

The output will be similar to like this:

----------------------------------
   Wed Apr 23 02:00:01 UTC 2014   
----------------------------------
Starting the Daily AMI creation: instance-23Apr14

AMI ID is: ami-d2xxxxx0



Looking for AMI older than 3 days:

 
Following AMI is found : ami-e65xxxxx4

Following are the snapshots associated with it : snap-axxxxxxf
snap-bxxxxxx2
snap-bxxxxxx5
 
Starting the Deregister of AMI... 

--------------------
|  DeregisterImage |
+---------+--------+
|  return |  true  |
+---------+--------+

Deleting the associated snapshots.... 

--------------------
|  DeleteSnapshot  |
+---------+--------+
|  return |  true  |
+---------+--------+
--------------------
|  DeleteSnapshot  |
+---------+--------+
|  return |  true  |
+---------+--------+
--------------------
|  DeleteSnapshot  |
+---------+--------+
|  return |  true  |
+---------+--------+

 Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.


Script to know the servers running in AWS auto scaling group..

Following script can be used to know the servers which are running under AWS auto scaling group..

#!/bin/bash
#Script to know thw servers running in AWS auto scaling group
#By Ravi Gadgil
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name autoscaling-group | grep i- | awk '{print $6}' > /tmp/instances.txt
echo "-----------------------------------------------------------------------------------------"
for inc in `cat /tmp/instances.txt`; do aws ec2 describe-instances --instance-ids $inc | grep -ir 'publicdns\|keyname'; echo "-----------------------------------------------------------------------------------------" ;done

In following script we are just looking for hostname and respective login key but it can be further increased according to your need by increasing the parameters in grep..

Following will be the output of the script output:

[root@server auto-data]# ./autoscaleserver.sh 
-----------------------------------------------------------------------------------------
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  ec2-XXXXXXX.ap-southeast-1.compute.amazonaws.com   |||
-----------------------------------------------------------------------------------------
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  ec2-XXXXXXX.ap-southeast-1.compute.amazonaws.com  |||
-----------------------------------------------------------------------------------------
|||  KeyName               |  test                                                   |||
|||  PublicDnsName         |  ec2-XXXXXXX.ap-southeast-1.compute.amazonaws.com  |||
-----------------------------------------------------------------------------------------
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  ec2-XXXXXXX.ap-southeast-1.compute.amazonaws.com  |||
-----------------------------------------------------------------------------------------
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  ec2-XXXXXXX.ap-southeast-1.compute.amazonaws.com  |||
-----------------------------------------------------------------------------------------
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  ec2-XXXXXXX.ap-southeast-1.compute.amazonaws.com  |||
-----------------------------------------------------------------------------------------

 Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.

 

How to install and Configure lsyncd

Lsyncd is a service which is used to sync data between the directories with in a server or between the servers. It is very useful tool if you want to keep redundancy between the servers. It has bit of delay while it copies data from source to target destination but its very simple to configure and light on server.

There should be password less authentication set between the servers so that they can sync data..

If your looking to get it configured it via yum following will be steps:


[root@server downloads]# yum -y install lua lua-devel pkgconfig gcc asciidoc
[root@server downloads]# yum --enablerepo=dag -y install lsyncd   # install from DAG

Create the configuration file:

[root@server downloads]# cp /usr/share/doc/lsyncd/lrsync.lua /etc/lsyncd.conf

[root@server downloads]# vi /etc/lsyncd.conf

Following entry needs to be done in the Configuration file:

settings = {
statusFile = "/tmp/lsyncd.stat",
statusInterval = 1,
}

sync{
default.rsync,
# source directory
source="/var/www/html/",
# destination Hostname or IP address:(the name set in rsyncd.conf)
target="10.0.0.33::site",
# rsync's options
rsyncOpts="-az",
# excluding list
excludeFrom="/etc/rsync_exclude.lst",
}

[root@server downloads]# /etc/rc.d/init.d/lsyncd start
Starting lsyncd: [  OK  }

[root@server downloads]# chkconfig lsyncd on


If your looking for source installation than following steps can be used:

# yum -y install lua lua-devel pkgconfig gcc asciidoc
# cd /tmp
# mkdir lsync_src; cd lsync_src; wget http://lsyncd.googlecode.com/files/lsyncd-2.1.4.tar.gz
# pwd
# tar zxf lsyncd-2.1.4.tar.gz; cd lsyncd-2.1.4
# ls
# export CFLAGS="-march=native -O2"
# ./configure && make && make install

Create the Configuration file:

[root@server downloads]# vi /etc/init.d/lsyncd

Content of file:

#!/bin/bash
#
# lsyncd: Starts the lsync Daemon
#
# chkconfig: 345 99 90
# description: Lsyncd uses rsync to synchronize local directories with a remote
# machine running rsyncd. Lsyncd watches multiple directories
# trees through inotify. The first step after adding the watches# is to, rsync all directories with the remote host, and then sync
# single file buy collecting the inotify events.
# processname: lsyncd

. /etc/rc.d/init.d/functions

config="/etc/lsyncd.lua"
lsyncd="/usr/local/bin/lsyncd"
lockfile="/var/lock/subsys/lsyncd"
pidfile="/var/run/lsyncd.pid"
prog="lsyncd"
RETVAL=0

start() {
    if [ -f $lockfile ]; then
        echo -n $"$prog is already running: "
        echo
        else
        echo -n $"Starting $prog: "
        daemon $lsyncd -pidfile $pidfile $config
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch $lockfile
        return $RETVAL
    fi
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $lsyncd

        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch $lockfile
        return $RETVAL
    fi
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $lsyncd
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f $lockfile
    return $RETVAL
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    status)
        status $lsyncd
        ;;
    *)
        echo "Usage: lsyncd {start|stop|restart|status}"
        exit 1
esac

exit $?

Give the executable permission to file:

[root@server downloads]# chmod 775 /etc/init.d/lsyncd

Make the main configuration file having details of data to be synced:

[root@server downloads]# vi /etc/lsyncd.lua

Content of the file with required parameters:

settings  {
        statusFile = "/tmp/lsyncd.stat",
        statusInterval = 1,
                pidfile = "/var/run/lsyncd.pid",
                logfile = "/var/log/lsyncd/lsyncd.log",
}

sync {
        default.rsync,
        source="/nfs-data",
        target="ip-10-10-10-10:/nfs-data",
        rsync = {
        compress = true,
        acls = true,
        verbose = true,
        owner = true,
        perms = true,
        group = true,
        rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"

        }
}

In source installation we can define the parameters which are needed for our sync like ownership, permissions, groups etc..

There are cases when lsyncsd stop working due to lack of open connection so use the following commands to increase the parameters:

[root@server downloads]# echo 100000 | sudo tee /proc/sys/fs/inotify/max_user_watches

[root@server downloads]# echo fs.inotify.max_user_watches=100000 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p

[root@ip-10-130-226-199 lsyncd]# rm /var/run/lsyncd.pid rm: remove regular file `/var/run/lsyncd.pid'? y
[root@ip-10-130-226-199 lsyncd]# rm /var/lock/subsys/lsyncd rm: remove regular empty file `/var/lock/subsys/lsyncd'? y    
[root@ip-10-130-226-199 lsyncd]# service lsyncd statuslsyncd is stopped
[root@ip-10-130-226-199 lsyncd]# service lsyncd start
Starting lsyncd:                                           [  OK  ]
[root@ip-10-130-226-199 lsyncd]# service lsyncd status
lsyncd (pid  23659) is running...
[root@ip-10-130-226-199 lsyncd]# view lsyncd.log 




 

Saturday, April 5, 2014

How to install and configure Google Page Speed Module for Nginx

Google Page speed module is very good tool to optimize sites performance by arranging CSS, images, java scripts etc in a better way which results in better performance from site..

Download the Page Speed Module and unzip it

[root@server downloads]# wget https://github.com/pagespeed/ngx_pagespeed/archive/release-1.7.30.4-beta.zip
[root@server downloads]# unzip release-1.7.30.4-beta
[root@server downloads]# cd ngx_pagespeed-release-1.7.30.4-beta/

We also need posl to make module work so download and untar it in page speed module directory..

[root@server downloads]# wget https://dl.google.com/dl/page-speed/psol/1.7.30.4.tar.gz
[root@server downloads]# tar -xzvf 1.7.30.4.tar.gz

Now compile the nginx with page speed module enable in it. Go in to the directory where you have your nginx setup available and configure it with including the page speed module.

[root@theteamie nginx-1.5.7]# ./configure --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --pid-path=/tmp/nginx/nginx.pid --user=apache --group=apache --with-http_ssl_module --with-pcre-jit --with-debug --with-http_realip_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_uwsgi_module --without-http_scgi_module --add-module=/root/downloads/nginx-modules/nginx-upstream-fair --add-module=/root/downloads/nginx-modules/nginx-upload-progress-module --add-module=/root/downloads/nginx-modules/ngx_devel_kit --add-module=/root/downloads/nginx-modules/lua-nginx-module --add-module=/root/downloads/nginx-modules/ngx_pagespeed-release-1.7.30.4-beta --with-http_stub_status_module
[root@theteamie nginx-1.5.7]# make ; make install
# Check whether its installed properly or not
[root@theteamie nginx-1.5.7]# /usr/sbin/nginx -V nginx version: nginx/1.5.7 built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --pid-path=/tmp/nginx/nginx.pid --user=apache --group=apache --with-http_ssl_module --with-pcre-jit --with-debug --with-http_realip_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_uwsgi_module --without-http_scgi_module --add-module=/root/downloads/nginx-modules/nginx-upstream-fair --add-module=/root/downloads/nginx-modules/nginx-upload-progress-module --add-module=/root/downloads/nginx-modules/ngx_devel_kit --add-module=/root/downloads/nginx-modules/lua-nginx-module --add-module=/root/downloads/nginx-modules/ngx_pagespeed-release-1.7.30.4-beta --with-http_stub_status_module

Go to vhost your nginx configuration to whom you want to include the page speed module and add the following lines in its server portion.

pagespeed on;
pagespeed FileCachePath /var/cache/nginx/pagespeed;
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
  add_header "" "";
}
location ~ "^/ngx_pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }
location /ngx_pagespeed_statistics {
  allow 127.0.0.1; deny all;
}
location /ngx_pagespeed_message {
  allow 127.0.0.1; deny all;
}

Create the /var/cache/nginx/pagespeed so that page speed module can store its data there..

[root@theteamie nginx-1.5.7]# mkdir -p /var/cache/nginx/pagespeed
[root@theteamie nginx-1.5.7]# chmod 777 /var/cache/nginx/pagespeed

Check the nginx configuration is correct than restart the nginx service..

[root@theteamie sites]# /usr/sbin/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@theteamie sites]# /usr/sbin/nginx restart

Now we are done and Page speed module is install successfully, Can check in your sites header which is configured to be working with page speed to see its working correctly or not..

 

Friday, April 4, 2014

Script to automatically create an AMI of server than add it to Auto Scaling Configuration and Group

Following script can help you to create an AMI of running server, than automatically create a auto scaling configuration from newly created AMI and updating the auto scaling group with new auto scaling configuration.

Note: To run this script AWS cli needs to be configured correctly and this script should be run in the server of whom you want to create AMI.

If in case your AWS cli is not configured follow the following steps..

Script:

#!/bin/bash
#Script to create ami of running instance, make launch Conf from it and than add it to Auto Scaling group
#By Ravi Gadgil

 #To get the current Instance ID
wget -q -O - http://169.254.169.254/latest/meta-data/instance-id > /tmp/instanceID.txt

 #To get the current Instance Type
wget -q -O - http://169.254.169.254/latest/meta-data/instance-type > /tmp/instanceType.txt

 #To create a unique AMI name for this script
echo "test-ami-`date +%d%b%y%H%M`" > /tmp/aminame.txt

 #To create a unique Launch Conf name for this script
echo "test-lau-`date +%d%b%y%H%M`" > /tmp/launchConf.txt

echo ""
echo "Starting the ami creation with name as `cat /tmp/aminame.txt`"
echo ""

 #Creating AMI of current server by taking above values
aws ec2 create-image --instance-id `cat /tmp/instanceID.txt` --name "`cat /tmp/aminame.txt`" --description "This is for testing auto AMI creation" --no-reboot | grep -ir ami | awk '{print $4}' > /tmp/amiID.txt

 #Showing the AMI name created by AWS
echo "AMI ID is `cat /tmp/amiID.txt`"

echo ""

 #Showing the parameters which will be used while creating the Launch Conf
echo "Creating the launch config as `cat /tmp/launchConf.txt` with key as mykey.pem Instance type `cat /tmp/instanceType.txt` and security group ravi-test"
echo ""

 #Creating the Launch Config with defining the key name to be used and security group
aws autoscaling create-launch-configuration --launch-configuration-name `cat /tmp/launchConf.txt` --image-id `cat /tmp/amiID.txt` --instance-type `cat /tmp/instanceType.txt` --key-name mykey --security-groups ravi-test --iam-instance-profile test1
echo "The launch Config created succesfully as `cat /tmp/launchConf.txt`"
echo ""
echo "Updating the Auto scaling Group test-prod-autoscaling with new launch Conf"

 #Updating the auto scaling group with new launch Conf
aws autoscaling update-auto-scaling-group --auto-scaling-group-name test-prod-autoscaling --launch-configuration-name `cat /tmp/launchConf.txt`

#some happy faces after all done well :)
echo "The Auto scaling group is updated succesfully...:)"

Note:  The AMI creation may take time depending upon the image type and EBS attached to it but all work besides that will be completed and once the AMI will be created all the process will be fully functional..:)


 

How to install and Configure Amazon Web Services(AWS) cli

Amazon Web Services(AWS) has its own set of command line interface tool which has all the necessary tools to run all AWS commands and is managed by them so updated frequently..

Download the zip folder containing all the required packages:

[root@server downloads]# wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
--2014-04-03 19:39:18--  https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
Resolving s3.amazonaws.com... 207.171.185.200
Connecting to s3.amazonaws.com|207.171.185.200|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6063815 (5.8M) [application/zip]
Saving to: “awscli-bundle.zip”
100%[=============================================================>] 6,063,815   1.52M/s   in 8.3s   
2014-04-03 19:39:28 (717 KB/s) - “awscli-bundle.zip” saved [6063815/6063815]

Unzip the folder downloaded:

[root@server downloads]# unzip awscli-bundle.zip 
Archive:  awscli-bundle.zip
  inflating: awscli-bundle/install   
  inflating: awscli-bundle/packages/argparse-1.2.1.tar.gz  
  inflating: awscli-bundle/packages/awscli-1.3.6.tar.gz  
  inflating: awscli-bundle/packages/bcdoc-0.12.2.tar.gz  
  inflating: awscli-bundle/packages/botocore-0.40.0.tar.gz  
  inflating: awscli-bundle/packages/colorama-0.2.5.tar.gz  
  inflating: awscli-bundle/packages/docutils-0.11.tar.gz  
  inflating: awscli-bundle/packages/jmespath-0.3.1.tar.gz  
  inflating: awscli-bundle/packages/ordereddict-1.1.tar.gz  
  inflating: awscli-bundle/packages/ply-3.4.tar.gz  
  inflating: awscli-bundle/packages/pyasn1-0.1.7.tar.gz  
  inflating: awscli-bundle/packages/python-dateutil-2.2.tar.gz  
  inflating: awscli-bundle/packages/rsa-3.1.2.tar.gz  
  inflating: awscli-bundle/packages/simplejson-3.3.0.tar.gz  
  inflating: awscli-bundle/packages/six-1.6.1.tar.gz  
  inflating: awscli-bundle/packages/virtualenv-1.10.1.tar.gz  

Go the unzipped folder:

[root@server downloads]# cd awscli-bundle
[root@server awscli-bundle]# ls
install  packages

Install the AWS cli to any destination where needed in this example we are installing it in /bin so that we don't have to include later in environment variable:

[root@server awscli-bundle]# ./install -b /bin/aws
Running cmd: /usr/bin/python virtualenv.py --python /usr/bin/python /root/.local/lib/aws
Running cmd: /root/.local/lib/aws/bin/pip install --no-index --find-links file:///root/downloads/awscli-bundle/packages awscli-1.3.6.tar.gz
You can now run: /bin/aws --version
[root@server awscli-bundle]# aws --version aws-cli/1.3.6 Python/2.6.8 Linux/3.2.39-6.88.amzn1.x86_64
[root@server awscli-bundle]# /bin/aws --version aws-cli/1.3.6 Python/2.6.8 Linux/3.2.39-6.88.amzn1.x86_64


How the AWS cli is needed to be configured with you IAM credentials so that it works correctly with permission user have:

To configure these you need your IAM Access Key ID or Secret Access Key..

[root@server scripts]# aws configure
AWS Access Key ID [None]: AXXXXXXXXXXXXXXXQ
AWS Secret Access Key [None]: 20XXXXXXXXXXXXXXXXXXXXXXXXXXXXST
Default region name [None]: ap-southeast-1
Default output format [None]: table

Enter the Default region name where your working and the output format in which you want your results to be.

There are three output formats text, table and json and output can be seen in different output format than default by adding --output json/text/table at the end of any AWS command.

 

Thursday, January 23, 2014

How to remove GlusterFS Volumes

To remove the GlusterFS Volume from the servers following steps need to be followed:

These Steps need to be followed on all the servers which are part of gluster Cluster:


[root@ip-10-138-150-225 ~]# setfattr -x trusted.glusterfs.volume-id /data/share
[root@ip-10-138-150-225 ~]# setfattr -x trusted.gfid /data/share
[root@ip-10-138-150-225 ~]# service glusterd stop
       [  OK  ]
[root@ip-10-138-150-225 ~]# cd /data/share/                [root@ip-10-138-150-225 share]# ls -a .  ..  a1  a2  a3  b1  b2  b3  c1  c2  c3  c4  d1  d2  d3  .glusterfs [root@ip-10-138-150-225 share]# rm -rf .glusterfs [root@ip-10-138-150-225 share]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4  d1  d2  d3 [root@ip-10-138-150-225 share]# cd /var/lib/glusterd/ [root@ip-10-138-150-225 glusterd]# ls glusterd.info  glustershd  groups  hooks  nfs  options  peers  vols [root@ip-10-138-150-225 glusterd]# rm -rf * [root@ip-10-138-150-225 glusterd]# service glusterd start Starting glusterd:                                         [  OK  ]

 

/data/share : Is the directory for which gluster is setup..
.glusterfs : Is the directory where all meta data is stored so it need to be removed..
/var/lib/glusterd/ : Directory where volume information is stored..

Note don't delete  the all files in /var/lib/glusterd/ if you have more than one gluster volume configured in server..


 

How to configure raid 0 in AWS with glusterFS to have high availability

Here is the procedure to configure Raid 0 on AWS's EBS to have high performance and GlusterFS to get High Availability..

This process can be used to get Central Storage in AWS as well as physical servers, as some of the Application needs central storage..

We are using 2 amazon instances with 4 EBS attached in each to configure Raid 0 on them to have good throughput..

Server 1 : ip-10-128-50-246
Server 2 : ip-10-138-150-225

Check the Attached EBS in each of the server:
 
[ec2-user@ip-10-128-50-246 ~]$ hostname
ip-10-128-50-246
[ec2-user@ip-10-128-50-246 ~]$ lsblk 
NAME  MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdb  202:16   0   5G  0 disk 
xvdc  202:32   0   5G  0 disk 
xvdd  202:48   0   5G  0 disk 
xvde  202:64   0   5G  0 disk 
xvda1 202:1    0   8G  0 disk /

[root@ip-10-138-150-225 ec2-user]# hostname 
ip-10-138-150-225
[root@ip-10-138-150-225 ec2-user]# lsblk 
NAME  MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvdb  202:16   0   5G  0 disk 
xvdc  202:32   0   5G  0 disk 
xvdd  202:48   0   5G  0 disk 
xvde  202:64   0   5G  0 disk 
xvda1 202:1    0   8G  0 disk /

Configure the Raid 0 in both the servers with disks available:

[root@ip-10-128-50-246 ec2-user]# mdadm --create /dev/md0 --level=0 --raid-devices=4 /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@ip-10-128-50-246 ec2-user]# mdadm --detail /dev/md0 /dev/md0:         Version : 1.2   Creation Time : Thu Jan 23 05:34:02 2014      Raid Level : raid0      Array Size : 20969472 (20.00 GiB 21.47 GB)    Raid Devices : 4   Total Devices : 4     Persistence : Superblock is persistent
    Update Time : Thu Jan 23 05:34:02 2014           State : clean  Active Devices : 4 Working Devices : 4  Failed Devices : 0   Spare Devices : 0
     Chunk Size : 512K
           Name : ip-10-128-50-246:0  (local to host ip-10-128-50-246)            UUID : 58fb9324:9aba25e7:db3a049d:525c8a98          Events : 0
    Number   Major   Minor   RaidDevice State        0     202       16        0      active sync   /dev/sdb        1     202       32        1      active sync   /dev/sdc        2     202       48        2      active sync   /dev/sdd        3     202       64        3      active sync   /dev/sde


[root@ip-10-138-150-225 ec2-user]# mdadm --create /dev/md0 --level=0 --raid-devices=4 /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@ip-10-138-150-225 ec2-user]# mdadm --detail /dev/md0 /dev/md0:         Version : 1.2   Creation Time : Thu Jan 23 05:35:28 2014      Raid Level : raid0      Array Size : 20969472 (20.00 GiB 21.47 GB)    Raid Devices : 4   Total Devices : 4     Persistence : Superblock is persistent
    Update Time : Thu Jan 23 05:35:28 2014           State : clean  Active Devices : 4 Working Devices : 4  Failed Devices : 0   Spare Devices : 0
     Chunk Size : 512K
           Name : ip-10-138-150-225:0  (local to host ip-10-138-150-225)            UUID : 85c9c957:d7d808d8:3a8344a2:9e994f83          Events : 0
    Number   Major   Minor   RaidDevice State        0     202       16        0      active sync   /dev/sdb        1     202       32        1      active sync   /dev/sdc        2     202       48        2      active sync   /dev/sdd        3     202       64        3      active sync   /dev/sde

 /dev/md0 : the single mount point for our Raid.


Format the mount point to desired File System, we are formatting it in EXT4:

[root@ip-10-128-50-246 ec2-user]# mkfs.ext4 /dev/md0
mke2fs 1.42.3 (14-May-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=512 blocks
1310720 inodes, 5242368 blocks
262118 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000
Allocating group tables: done                            Writing inode tables: done                            Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done  


[root@ip-10-138-150-225 ec2-user]# mkfs.ext4 /dev/md0
mke2fs 1.42.3 (14-May-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=512 blocks
1310720 inodes, 5242368 blocks
262118 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000
Allocating group tables: done                            Writing inode tables: done                            Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done  

 Mount the Raid mount point to desired directory:

[root@ip-10-128-50-246 ec2-user]# mkdir /data
[root@ip-10-128-50-246 ec2-user]# mount /dev/md0 /data
[root@ip-10-128-50-246 ec2-user]# df -h /data/
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               20G  172M   19G   1% /data

[root@ip-10-138-150-225 ec2-user]# mkdir /data
[root@ip-10-138-150-225 ec2-user]# mount /dev/md0 /data
[root@ip-10-138-150-225 ec2-user]# df -h /data
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               20G  172M   19G   1% /data


Now configure the GlusterFS for /data directory for each of the servers:
To get full details of GlusterFS configure please follow following post http://simplyopensource.blogspot.com/2013/11/how-to-implement-glusterfs-in-amazon.html

[root@ip-10-128-50-246 ec2-user]# wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo; sed -i 's/$releasever/6/g' /etc/yum.repos.d/glusterfs-epel.repo ; yum install libibverbs-devel fuse-devel -y ; yum install -y glusterfs{-fuse,-server} ; service glusterd start ; modprobe fuse

You may have similar error while creating Gluster for /data as Gluster can't be implemented on Raid parent directory so we have created a sub directory /data/share and configured Gluster for it..


[root@ip-10-128-50-246 ec2-user]# gluster peer probe  ec2-54-255-3-126.ap-southeast-1.compute.amazonaws.com
peer probe: success
[root@ip-10-128-50-246 ec2-user]# gluster volume create Test-Volume replica 2 transport tcp ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:/data ec2-54-255-3-126.ap-southeast-1.compute.amazonaws.com:/data volume create: Test-Volume: failed: The brick ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:/data is a mount point. Please create a sub-directory under the mount point and use that as the brick directory. Or use 'force' at the end of the command if you want to override this behavior.
[root@ip-10-128-50-246 ec2-user]# mkdir -p /data/share
[root@ip-10-128-50-246 ec2-user]# gluster volume create Test-Volume replica 2 transport tcp ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:/data/share ec2-54-255-3-126.ap-southeast-1.compute.amazonaws.com:/data/share volume create: Test-Volume: success: please start the volume to access data
[root@ip-10-128-50-246 ec2-user]# gluster volume start Test-Volume volume start: Test-Volume: success
[root@ip-10-128-50-246 ec2-user]# gluster volume info   Volume Name: Test-Volume Type: Replicate Volume ID: 47477eaa-423b-4200-9f55-2966ae079a79 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:/data/share

Configure the glusterFS for /data/share and start the volume..

Mount the Gluster volume to desired directory to make single Data access point on both the servers:

[root@ip-10-128-50-246 ec2-user]# mount -t glusterfs ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:Test-Volume /mnt
[root@ip-10-128-50-246 ec2-user]# df -h /mnt
Filesystem            Size  Used Avail Use% Mounted on
ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:Test-Volume
                       20G  172M   19G   1% /mnt

[root@ip-10-138-150-225 ec2-user]# mount -t glusterfs ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:Test-Volume /mnt
[root@ip-10-138-150-225 ec2-user]# df -h /mnt
Filesystem            Size  Used Avail Use% Mounted on
ec2-122-248-213-231.ap-southeast-1.compute.amazonaws.com:Test-Volume
                       20G  172M   19G   1% /mnt

Now checking is our Gluster working good or not:


[root@ip-10-138-150-225 mnt]# touch a1 a2 a3
[root@ip-10-138-150-225 mnt]# ls
a1  a2  a3
[root@ip-10-128-50-246 ec2-user]# cd /mnt [root@ip-10-128-50-246 mnt]# ls a1  a2  a3 [root@ip-10-128-50-246 mnt]# touch b1 b2 b3 [root@ip-10-128-50-246 mnt]# ls a1  a2  a3  b1  b2  b3

Now lets try to break our mount and see if Gluster is working fine or not via un mounting the /data directory in one of the servers:

[root@ip-10-128-50-246 mnt]# umount -l /data
[root@ip-10-128-50-246 mnt]# cd /mnt [root@ip-10-128-50-246 mnt]# ls a1  a2  a3  b1  b2  b3 [root@ip-10-128-50-246 mnt]# touch c1 c2 c3 c4 [root@ip-10-128-50-246 mnt]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4
[root@ip-10-138-150-225 mnt]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4 [root@ip-10-138-150-225 mnt]# cd /data/share/ [root@ip-10-138-150-225 share]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4

Now lets mount back the directory to check we have consistent data:

[root@ip-10-128-50-246 mnt]# mount /dev/md0 /data
[root@ip-10-128-50-246 share]# pwd /data/share [root@ip-10-128-50-246 share]# ls -l total 0 -rw-r--r-- 2 root root 0 Jan 23 05:52 a1 -rw-r--r-- 2 root root 0 Jan 23 05:52 a2 -rw-r--r-- 2 root root 0 Jan 23 05:52 a3 -rw-r--r-- 2 root root 0 Jan 23 05:53 b1 -rw-r--r-- 2 root root 0 Jan 23 05:53 b2 -rw-r--r-- 2 root root 0 Jan 23 05:53 b3 [root@ip-10-128-50-246 share]# touch d1 d2 d3 [root@ip-10-128-50-246 share]# ls a1  a2  a3  b1  b2  b3  d1  d2  d3 [root@ip-10-128-50-246 share]# cd /mnt/ [root@ip-10-128-50-246 mnt]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4  d1  d2  d3 [root@ip-10-128-50-246 mnt]# cd /data/share [root@ip-10-128-50-246 share]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4  d1  d2  d3

[root@ip-10-138-150-225 share]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4  d1  d2  d3 [root@ip-10-138-150-225 share]# cd /data/share [root@ip-10-138-150-225 share]# ls a1  a2  a3  b1  b2  b3  c1  c2  c3  c4  d1  d2  d3

The data replicated very well across the servers and the data also gets updated in remounted directory as soon as we create any new content in servers..
 

Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...