Thursday, February 25, 2016

Script to monitor sites via response code

To find out the availability of our sites following script can be very helpful as it will enable us to check the status code of sites. I have created the file which has list of sites which we have to monitor and than passing that file in script to monitor them.

First create a file having names of the sites which need to be monitored listed:
 [root@ip-10-0-1-55 ravi]# cat sitelist.txt 
https://www.theteamie.com
https://nyp-trial.theteamie.com
http://samsung.theteamie.com

Than use the following script to monitor the sites and check the return status.
Note : I am only considering status code 200 and 301 as acceptable return code rest all are consider as error.
#!/bin/bash
#By Ravi Gadgil
#Script to monitor sites using there return status

for i in `cat /home/ravi/sitelist.txt`;
do
echo -e "-----------------------------------------------------------------"
echo -e "$i is being checked"
res=`curl -I -s $i | grep HTTP/1.1 | awk {'print $2'}`
echo -e "$res"
    if [ $res -ne 200 ] && [ $res -ne 301 ];
        then
        echo "Error $res on $i"
        echo -e "----------------------------------------\nSiteName : $i \nStatus Code Returned : $res \n" >> /home/ravi/siteissue.txt

        else
        echo -e "$i is OK"
    fi
done

if [ -f /home/ravi/siteissue.txt ]
then
    echo -e "\n\n\nSites having issues:\n"
    cat /home/ravi/siteissue.txt
    rm -rf /home/ravi/siteissue.txt
fi


The output of the script will be in following way:
[root@ip-10-0-1-55 ravi]# ./sitescheck.sh 
-----------------------------------------------------------------
https://www.theteamie.com is being checked
301
https://www.theteamie.com is OK
-----------------------------------------------------------------
https://nyp-trial.theteamie.com is being checked
503
Error 503 on https://nyp-trial.theteamie.com
-----------------------------------------------------------------
http://samsung.theteamie.com is being checked
504
Error 504 on http://samsung.theteamie.com



Sites having issues:

----------------------------------------
SiteName : https://nyp-trial.theteamie.com 
Status Code Returned : 503 

----------------------------------------
SiteName : http://samsung.theteamie.com 
Status Code Returned : 504 


Use the siteissue.txt to mail you the effected sites if any is there.


How to use crontab with examples

Crontab is one of the most useful services in Linux which helps us to automate the tasks. It enables us to run commands on a specific interval of time.

It takes following 5 parameters into consideration:
# Minute   Hour   Day of Month       Month          Day of Week        Command    
# (0-59)  (0-23)     (1-31)    (1-12 or Jan-Dec)  (0-6 or Sun-Sat)              

 .---------------- minute (0 - 59) 
 |  .------------- hour (0 - 23)
 |  |  .---------- day of month (1 - 31)
 |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ... 
 |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)  OR sun,mon,tue,wed,thu,fri,sat 
 |  |  |  |  |
 *  *  *  *  *  <command to be executed>  

To edit cron for current user:
# crontab -e

To list cron for current user:
# crontab -l

To edit cron for any specific user:
# crontab -u ravi -e

To list cron for any specific user:
# crontab -u ravi -l

There are few predefined strings which can help to run cron at specific time frame:
string         meaning
------         -------
@reboot        Run once, at startup.
@yearly        Run once a year, "0 0 1 1 *".
@annually      (same as @yearly)
@monthly       Run once a month, "0 0 1 * *".
@weekly        Run once a week, "0 0 * * 0".
@daily         Run once a day, "0 0 * * *".
@midnight      (same as @daily)
@hourly        Run once an hour, "0 * * * *".

Examples to run crontab:

To run cron at every boot of system:
@reboot   /usr/scripts/test.sh

To run cron at every minute:
* * * * * /usr/scripts/test.sh

To run cron at every 5 minutes:
 */5 * * * * /usr/scripts/test.sh

To run cron on specific minutes(It will run on every 2 and 5 minutes of hours):
2,5 * * * *  /usr/scripts/test.sh

To run con every hour(It will run on every hour at 0 minute):
0 * * * *  /usr/scripts/test.sh

To run cron on every 2 hours:
* */2 * * * /usr/scripts/test.sh

To run cron for within specific interval of time(It will run from 3AM to 9PM):
* 3-21 * * *  /usr/scripts/test.sh

To run cron for specific time of year(It will run on 3AM and 5AM on 20th July):
0 3,5 20 7 *  /usr/scripts/test.sh

To run cron on any specific weekday(It will run on 1AM on every Sunday):
0 1 * * 0  /usr/scripts/test.sh




Wednesday, February 24, 2016

How to find daily Web server hits count

If we are hosting the Web servers in shared hosting or dedicated hosting and what to know the hits count sorted in descending order than following script can be very helpful. We are analyzing of access logs situated in /var/log/nginx folder to get the daily hits counts.

If you want to get the daily hits count the best time would be just before your log rotation happens as after the log rotation the access logs will become null and new logs will start to add in.

Script:
#!/bin/bash
#By Ravi Gadgil.
#Script to find daily hits count via web server access logs.

echo -e "Hits \t  Url"

for i in `find /var/log/nginx/ -name "*access.log"`;do echo $i | sed -e 's/_access.log/ /g' | sed -e 's/:/ /g'| cut -d'/' -f5 --output-delimiter=' ' | awk '{printf $0}'; grep -v 'jpg\|png\|jpeg\|gif\|js' $i | grep -ircn `date | awk ' { print $3 } '`; done | awk ' { print $2,"\011",$1 } ' | sort -nr 

Note: Access logs in my case is domain.com_access.log and not counting the "jpg,png,jpeg,gif,js" so if you want to count them as well remove the grep -v from command.

Wednesday, February 10, 2016

Script to delete Server with attached EBS volumes.

Following script is to delete Server with all attached EBS volumes with it. In AWS it can be tough task to remove Servers if it has EBS volumes attached in it and they need to be removed manually so following script can be a great help at that time.

Note : In order to make it work you need to have your AWS output set to table form else you need to do bit of changes as per your output type.

Script:
#!/bin/bash
#By Ravi Gadgil.
#Script to delete ami with attached snapshots.

#Take input of server to be deleted.
echo -e "$1" > /tmp/imageid.txt

#Find EBS associated with server.
aws ec2 describe-instances --instance-ids `cat /tmp/imageid.txt` | grep vol | awk ' { print $4 }' > /tmp/vol.txt


echo -e "Following are the volume associated with it : `cat /tmp/vol.txt`:\n "
 
echo -e "Starting the termination of Server... \n"

#Terminating server 
aws ec2 terminate-instances --instance-ids `cat /tmp/imageid.txt`

echo -e "\nDeleting the associated Volumes.... \n"

#Deleting Volumes associated with Server
for i in `cat /tmp/vol.txt`;do aws ec2 delete-volume --volume-id $i ; done

Pass the Server id as a parameter to script to see the magic..

Output:
[root@ip-10-0-1-55 ravi]# ./instanceremove.sh i-3fa5f317
Following are the volume associated with it : vol-514d0159
vol-93f2a39b
vol-81f5a489:
 
Starting the termination of Server... 

----------------------------
|    TerminateInstances    |
+--------------------------+
||  TerminatingInstances  ||
|+------------------------+|
||       InstanceId       ||
|+------------------------+|
||  i-3fa5f317            ||
|+------------------------+|
|||     CurrentState     |||
||+-------+--------------+||
||| Code  |    Name      |||
||+-------+--------------+||
|||  48   |  terminated  |||
||+-------+--------------+||
|||     PreviousState    |||
||+--------+-------------+||
|||  Code  |    Name     |||
||+--------+-------------+||
|||  80    |  stopped    |||
||+--------+-------------+||

Deleting the associated Volumes.... 


Script to delete AMI with attached snapshots.

Following script is to delete AMI with all attached snapshots with it. In AWS it can be tough task to remove AMI if it has snapshots attached in it and they need to be removed manually so following script can be a great help at that time.

Note : In order to make it work you need to have your AWS output set to table form else you need to do bit of changes as per your output type.

Script :

#!/bin/bash
#By Ravi Gadgil.
#Script to delete ami with attached snapshots.

#Take input of AMI to be deleted.
echo -e "$1" > /tmp/imageid.txt

#Find snapshots associated with AMI.
aws ec2 describe-images --image-ids `cat /tmp/imageid.txt` | grep snap | awk ' { print $4 }' > /tmp/snap.txt

echo -e "Following are the snapshots associated with it : `cat /tmp/snap.txt`:\n "
 
 echo -e "Starting the Deregister of AMI... \n"

#Deregistering the AMI 
aws ec2 deregister-image --image-id `cat /tmp/imageid.txt`

echo -e "\nDeleting the associated snapshots.... \n"

#Deleting snapshots attached to AMI
for i in `cat /tmp/snap.txt`;do aws ec2 delete-snapshot --snapshot-id $i ; done

Pass the AMI id as a parameter to script to see the magic..

Script output:
[root@ip-10-0-1-55 ravi]# ./amiremove.sh ami-34d08d66
Following are the snapshots associated with it : snap-cba6b326
snap-c4a6b329
snap-c1a6b32c
snap-c2a6b32f:
 
Starting the Deregister of AMI... 


Deleting the associated snapshots.... 




Change user IAM password with AWS CLI.

In order to change the password of IAM user in AWS following commands can be used.

First we need to create a json file having old and new password of the user.
[root@ip-10-0-1-55 ravi]# cat change.json 
{
    "OldPassword": "Ravi@123",
    "NewPassword": "Ravi@1234"
}

Following command will reset the password of user from which the command has been run.
[root@ip-10-0-1-55 ravi]# aws iam change-password --cli-input-json file://change.json

Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...