Wednesday, April 23, 2014

Script to create daily AMI of AWS server and delete AMI older than n no of days..

Following Script can be used to create AMI of AWS server daily to keep the backup and delete the AMI older than n no of days including snapshots associated with it..

#Script to create AMI of server on daily basis and deleting AMI older than n no of days
#By Ravi Gadgil

echo -e "----------------------------------\n   `date`   \n----------------------------------"

#To create a unique AMI name for this script
echo "instance-`date +%d%b%y`" > /tmp/aminame.txt

echo -e "Starting the Daily AMI creation: `cat /tmp/aminame.txt`\n"

#To create AMI of defined instance
aws ec2 create-image --instance-id i-1xxxxx34 --name "`cat /tmp/aminame.txt`" --description "This is for Daily auto AMI creation" --no-reboot | grep -i ami | awk '{print $4}' > /tmp/amiID.txt

#Showing the AMI name created by AWS
echo -e "AMI ID is: `cat /tmp/amiID.txt`\n"

echo -e "Looking for AMI older than 3 days:\n "

#Finding AMI older than 3 days which needed to be removed
echo "instance-`date +%d%b%y --date '4 days ago'`" > /tmp/amidel.txt

#Finding Image ID of instance which needed to be Deregistered
aws ec2 describe-images --filters "Name=name,Values=`cat /tmp/amidel.txt`" | grep -i imageid | awk '{ print  $4 }' > /tmp/imageid.txt

if [[ -s /tmp/imageid.txt ]];

echo -e "Following AMI is found : `cat /tmp/imageid.txt`\n"

#Find the snapshots attached to the Image need to be Deregister
aws ec2 describe-images --image-ids `cat /tmp/imageid.txt` | grep snap | awk ' { print $4 }' > /tmp/snap.txt

echo -e "Following are the snapshots associated with it : `cat /tmp/snap.txt`:\n "
echo -e "Starting the Deregister of AMI... \n"

#Deregistering the AMI 
aws ec2 deregister-image --image-id `cat /tmp/imageid.txt`

echo -e "\nDeleting the associated snapshots.... \n"

#Deleting snapshots attached to AMI
for i in `cat /tmp/snap.txt`;do aws ec2 delete-snapshot --snapshot-id $i ; done


echo -e "No AMI found older than minimum required no of days"


The output will be similar to like this:

   Wed Apr 23 02:00:01 UTC 2014   
Starting the Daily AMI creation: instance-23Apr14

AMI ID is: ami-d2xxxxx0

Looking for AMI older than 3 days:

Following AMI is found : ami-e65xxxxx4

Following are the snapshots associated with it : snap-axxxxxxf
Starting the Deregister of AMI... 

|  DeregisterImage |
|  return |  true  |

Deleting the associated snapshots.... 

|  DeleteSnapshot  |
|  return |  true  |
|  DeleteSnapshot  |
|  return |  true  |
|  DeleteSnapshot  |
|  return |  true  |

 Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.

Script to know the servers running in AWS auto scaling group..

Following script can be used to know the servers which are running under AWS auto scaling group..

#Script to know thw servers running in AWS auto scaling group
#By Ravi Gadgil
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name autoscaling-group | grep i- | awk '{print $6}' > /tmp/instances.txt
echo "-----------------------------------------------------------------------------------------"
for inc in `cat /tmp/instances.txt`; do aws ec2 describe-instances --instance-ids $inc | grep -ir 'publicdns\|keyname'; echo "-----------------------------------------------------------------------------------------" ;done

In following script we are just looking for hostname and respective login key but it can be further increased according to your need by increasing the parameters in grep..

Following will be the output of the script output:

[root@server auto-data]# ./ 
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |   |||
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  |||
|||  KeyName               |  test                                                   |||
|||  PublicDnsName         |  |||
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  |||
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  |||
|||  KeyName               |  test                                                  |||
|||  PublicDnsName         |  |||

 Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.


How to install and Configure lsyncd

Lsyncd is a service which is used to sync data between the directories with in a server or between the servers. It is very useful tool if you want to keep redundancy between the servers. It has bit of delay while it copies data from source to target destination but its very simple to configure and light on server.

There should be password less authentication set between the servers so that they can sync data..

If your looking to get it configured it via yum following will be steps:

[root@server downloads]# yum -y install lua lua-devel pkgconfig gcc asciidoc
[root@server downloads]# yum --enablerepo=dag -y install lsyncd   # install from DAG

Create the configuration file:

[root@server downloads]# cp /usr/share/doc/lsyncd/lrsync.lua /etc/lsyncd.conf

[root@server downloads]# vi /etc/lsyncd.conf

Following entry needs to be done in the Configuration file:

settings = {
statusFile = "/tmp/lsyncd.stat",
statusInterval = 1,

# source directory
# destination Hostname or IP address:(the name set in rsyncd.conf)
# rsync's options
# excluding list

[root@server downloads]# /etc/rc.d/init.d/lsyncd start
Starting lsyncd: [  OK  }

[root@server downloads]# chkconfig lsyncd on

If your looking for source installation than following steps can be used:

# yum -y install lua lua-devel pkgconfig gcc asciidoc
# cd /tmp
# mkdir lsync_src; cd lsync_src; wget
# pwd
# tar zxf lsyncd-2.1.4.tar.gz; cd lsyncd-2.1.4
# ls
# export CFLAGS="-march=native -O2"
# ./configure && make && make install

Create the Configuration file:

[root@server downloads]# vi /etc/init.d/lsyncd

Content of file:

# lsyncd: Starts the lsync Daemon
# chkconfig: 345 99 90
# description: Lsyncd uses rsync to synchronize local directories with a remote
# machine running rsyncd. Lsyncd watches multiple directories
# trees through inotify. The first step after adding the watches# is to, rsync all directories with the remote host, and then sync
# single file buy collecting the inotify events.
# processname: lsyncd

. /etc/rc.d/init.d/functions


start() {
    if [ -f $lockfile ]; then
        echo -n $"$prog is already running: "
        echo -n $"Starting $prog: "
        daemon $lsyncd -pidfile $pidfile $config
        [ $RETVAL = 0 ] && touch $lockfile
        return $RETVAL

stop() {
    echo -n $"Stopping $prog: "
    killproc $lsyncd

        [ $RETVAL = 0 ] && touch $lockfile
        return $RETVAL

stop() {
    echo -n $"Stopping $prog: "
    killproc $lsyncd
    [ $RETVAL = 0 ] && rm -f $lockfile
    return $RETVAL

case "$1" in
        status $lsyncd
        echo "Usage: lsyncd {start|stop|restart|status}"
        exit 1

exit $?

Give the executable permission to file:

[root@server downloads]# chmod 775 /etc/init.d/lsyncd

Make the main configuration file having details of data to be synced:

[root@server downloads]# vi /etc/lsyncd.lua

Content of the file with required parameters:

settings  {
        statusFile = "/tmp/lsyncd.stat",
        statusInterval = 1,
                pidfile = "/var/run/",
                logfile = "/var/log/lsyncd/lsyncd.log",

sync {
        rsync = {
        compress = true,
        acls = true,
        verbose = true,
        owner = true,
        perms = true,
        group = true,
        rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"


In source installation we can define the parameters which are needed for our sync like ownership, permissions, groups etc..

There are cases when lsyncsd stop working due to lack of open connection so use the following commands to increase the parameters:

[root@server downloads]# echo 100000 | sudo tee /proc/sys/fs/inotify/max_user_watches

[root@server downloads]# echo fs.inotify.max_user_watches=100000 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p

[root@ip-10-130-226-199 lsyncd]# rm /var/run/ rm: remove regular file `/var/run/'? y
[root@ip-10-130-226-199 lsyncd]# rm /var/lock/subsys/lsyncd rm: remove regular empty file `/var/lock/subsys/lsyncd'? y    
[root@ip-10-130-226-199 lsyncd]# service lsyncd statuslsyncd is stopped
[root@ip-10-130-226-199 lsyncd]# service lsyncd start
Starting lsyncd:                                           [  OK  ]
[root@ip-10-130-226-199 lsyncd]# service lsyncd status
lsyncd (pid  23659) is running...
[root@ip-10-130-226-199 lsyncd]# view lsyncd.log 


Saturday, April 5, 2014

How to install and configure Google Page Speed Module for Nginx

Google Page speed module is very good tool to optimize sites performance by arranging CSS, images, java scripts etc in a better way which results in better performance from site..

Download the Page Speed Module and unzip it

[root@server downloads]# wget
[root@server downloads]# unzip release-
[root@server downloads]# cd ngx_pagespeed-release-

We also need posl to make module work so download and untar it in page speed module directory..

[root@server downloads]# wget
[root@server downloads]# tar -xzvf

Now compile the nginx with page speed module enable in it. Go in to the directory where you have your nginx setup available and configure it with including the page speed module.

[root@theteamie nginx-1.5.7]# ./configure --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --pid-path=/tmp/nginx/ --user=apache --group=apache --with-http_ssl_module --with-pcre-jit --with-debug --with-http_realip_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_uwsgi_module --without-http_scgi_module --add-module=/root/downloads/nginx-modules/nginx-upstream-fair --add-module=/root/downloads/nginx-modules/nginx-upload-progress-module --add-module=/root/downloads/nginx-modules/ngx_devel_kit --add-module=/root/downloads/nginx-modules/lua-nginx-module --add-module=/root/downloads/nginx-modules/ngx_pagespeed-release- --with-http_stub_status_module
[root@theteamie nginx-1.5.7]# make ; make install
# Check whether its installed properly or not
[root@theteamie nginx-1.5.7]# /usr/sbin/nginx -V nginx version: nginx/1.5.7 built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/sbin/nginx --pid-path=/tmp/nginx/ --user=apache --group=apache --with-http_ssl_module --with-pcre-jit --with-debug --with-http_realip_module --with-http_gzip_static_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_uwsgi_module --without-http_scgi_module --add-module=/root/downloads/nginx-modules/nginx-upstream-fair --add-module=/root/downloads/nginx-modules/nginx-upload-progress-module --add-module=/root/downloads/nginx-modules/ngx_devel_kit --add-module=/root/downloads/nginx-modules/lua-nginx-module --add-module=/root/downloads/nginx-modules/ngx_pagespeed-release- --with-http_stub_status_module

Go to vhost your nginx configuration to whom you want to include the page speed module and add the following lines in its server portion.

pagespeed on;
pagespeed FileCachePath /var/cache/nginx/pagespeed;
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
  add_header "" "";
location ~ "^/ngx_pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }
location /ngx_pagespeed_statistics {
  allow; deny all;
location /ngx_pagespeed_message {
  allow; deny all;

Create the /var/cache/nginx/pagespeed so that page speed module can store its data there..

[root@theteamie nginx-1.5.7]# mkdir -p /var/cache/nginx/pagespeed
[root@theteamie nginx-1.5.7]# chmod 777 /var/cache/nginx/pagespeed

Check the nginx configuration is correct than restart the nginx service..

[root@theteamie sites]# /usr/sbin/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@theteamie sites]# /usr/sbin/nginx restart

Now we are done and Page speed module is install successfully, Can check in your sites header which is configured to be working with page speed to see its working correctly or not..


Friday, April 4, 2014

Script to automatically create an AMI of server than add it to Auto Scaling Configuration and Group

Following script can help you to create an AMI of running server, than automatically create a auto scaling configuration from newly created AMI and updating the auto scaling group with new auto scaling configuration.

Note: To run this script AWS cli needs to be configured correctly and this script should be run in the server of whom you want to create AMI.

If in case your AWS cli is not configured follow the following steps..


#Script to create ami of running instance, make launch Conf from it and than add it to Auto Scaling group
#By Ravi Gadgil

 #To get the current Instance ID
wget -q -O - > /tmp/instanceID.txt

 #To get the current Instance Type
wget -q -O - > /tmp/instanceType.txt

 #To create a unique AMI name for this script
echo "test-ami-`date +%d%b%y%H%M`" > /tmp/aminame.txt

 #To create a unique Launch Conf name for this script
echo "test-lau-`date +%d%b%y%H%M`" > /tmp/launchConf.txt

echo ""
echo "Starting the ami creation with name as `cat /tmp/aminame.txt`"
echo ""

 #Creating AMI of current server by taking above values
aws ec2 create-image --instance-id `cat /tmp/instanceID.txt` --name "`cat /tmp/aminame.txt`" --description "This is for testing auto AMI creation" --no-reboot | grep -ir ami | awk '{print $4}' > /tmp/amiID.txt

 #Showing the AMI name created by AWS
echo "AMI ID is `cat /tmp/amiID.txt`"

echo ""

 #Showing the parameters which will be used while creating the Launch Conf
echo "Creating the launch config as `cat /tmp/launchConf.txt` with key as mykey.pem Instance type `cat /tmp/instanceType.txt` and security group ravi-test"
echo ""

 #Creating the Launch Config with defining the key name to be used and security group
aws autoscaling create-launch-configuration --launch-configuration-name `cat /tmp/launchConf.txt` --image-id `cat /tmp/amiID.txt` --instance-type `cat /tmp/instanceType.txt` --key-name mykey --security-groups ravi-test --iam-instance-profile test1
echo "The launch Config created succesfully as `cat /tmp/launchConf.txt`"
echo ""
echo "Updating the Auto scaling Group test-prod-autoscaling with new launch Conf"

 #Updating the auto scaling group with new launch Conf
aws autoscaling update-auto-scaling-group --auto-scaling-group-name test-prod-autoscaling --launch-configuration-name `cat /tmp/launchConf.txt`

#some happy faces after all done well :)
echo "The Auto scaling group is updated succesfully...:)"

Note:  The AMI creation may take time depending upon the image type and EBS attached to it but all work besides that will be completed and once the AMI will be created all the process will be fully functional..:)


How to install and Configure Amazon Web Services(AWS) cli

Amazon Web Services(AWS) has its own set of command line interface tool which has all the necessary tools to run all AWS commands and is managed by them so updated frequently..

Download the zip folder containing all the required packages:

[root@server downloads]# wget
--2014-04-03 19:39:18--
Connecting to||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6063815 (5.8M) [application/zip]
Saving to: “”
100%[=============================================================>] 6,063,815   1.52M/s   in 8.3s   
2014-04-03 19:39:28 (717 KB/s) - “” saved [6063815/6063815]

Unzip the folder downloaded:

[root@server downloads]# unzip 
  inflating: awscli-bundle/install   
  inflating: awscli-bundle/packages/argparse-1.2.1.tar.gz  
  inflating: awscli-bundle/packages/awscli-1.3.6.tar.gz  
  inflating: awscli-bundle/packages/bcdoc-0.12.2.tar.gz  
  inflating: awscli-bundle/packages/botocore-0.40.0.tar.gz  
  inflating: awscli-bundle/packages/colorama-0.2.5.tar.gz  
  inflating: awscli-bundle/packages/docutils-0.11.tar.gz  
  inflating: awscli-bundle/packages/jmespath-0.3.1.tar.gz  
  inflating: awscli-bundle/packages/ordereddict-1.1.tar.gz  
  inflating: awscli-bundle/packages/ply-3.4.tar.gz  
  inflating: awscli-bundle/packages/pyasn1-0.1.7.tar.gz  
  inflating: awscli-bundle/packages/python-dateutil-2.2.tar.gz  
  inflating: awscli-bundle/packages/rsa-3.1.2.tar.gz  
  inflating: awscli-bundle/packages/simplejson-3.3.0.tar.gz  
  inflating: awscli-bundle/packages/six-1.6.1.tar.gz  
  inflating: awscli-bundle/packages/virtualenv-1.10.1.tar.gz  

Go the unzipped folder:

[root@server downloads]# cd awscli-bundle
[root@server awscli-bundle]# ls
install  packages

Install the AWS cli to any destination where needed in this example we are installing it in /bin so that we don't have to include later in environment variable:

[root@server awscli-bundle]# ./install -b /bin/aws
Running cmd: /usr/bin/python --python /usr/bin/python /root/.local/lib/aws
Running cmd: /root/.local/lib/aws/bin/pip install --no-index --find-links file:///root/downloads/awscli-bundle/packages awscli-1.3.6.tar.gz
You can now run: /bin/aws --version
[root@server awscli-bundle]# aws --version aws-cli/1.3.6 Python/2.6.8 Linux/3.2.39-6.88.amzn1.x86_64
[root@server awscli-bundle]# /bin/aws --version aws-cli/1.3.6 Python/2.6.8 Linux/3.2.39-6.88.amzn1.x86_64

How the AWS cli is needed to be configured with you IAM credentials so that it works correctly with permission user have:

To configure these you need your IAM Access Key ID or Secret Access Key..

[root@server scripts]# aws configure
Default region name [None]: ap-southeast-1
Default output format [None]: table

Enter the Default region name where your working and the output format in which you want your results to be.

There are three output formats text, table and json and output can be seen in different output format than default by adding --output json/text/table at the end of any AWS command.


Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...