Wednesday, December 13, 2017

How to copy RDS snapshot to different region and delete older

Following script can be used to copy the AWS RDS snapshot to different region on daily basis and delete the older than n no of days.

#!/bin/bash
#Script to copy daily DB snapshot to US region and delete older than 5 days
#By Ravi Gadgil

echo -e "----------------------------------\n   `date`   \n----------------------------------"


echo -e "Finding the DB Snapshot ID to be copied."
snapname=$(echo -e "productiondb-`date +%F`-19-04")
newsnapname=$(echo -e "productiondb-`date +%F`-19-04-us")
echo -e "DB Snapshot which is being copied $snapname "

echo -e "Starting the copy process as $newsnapname"
aws rds copy-db-snapshot --source-db-snapshot-identifier arn:aws:rds:ap-southeast-1:52xxxxxxxx33:snapshot:rds:$snapname --target-db-snapshot-identifier $newsnapname --copy-tags --region us-east-1


echo -e "Looking for DB SnapShot older than 5 days:\n "

#Finding SnapShot older than 5 days which needed to be removed
echo "productiondb-`date +%F --date '5 days ago'`-19-04-us" > /tmp/dbsnapdel.txt

if [[ -s /tmp/dbsnapdel.txt ]];
then

echo -e "Following SnapShot is found : `cat /tmp/dbsnapdel.txt`\n"

echo -e "\nDeleting the associated snapshots.... \n"

#Deleting snapshots of Older days
aws rds delete-db-snapshot --db-snapshot-identifier `cat /tmp/dbsnapdel.txt` --region us-east-1

else

echo -e "No SnapShot found older than minimum required no of days"
fi

The output will be similar to this:

----------------------------------
   Tue Dec 12 20:30:02 UTC 2017   
----------------------------------
Finding the DB Snapshot ID to be copied.
DB Snapshot which is being copied productiondb-2017-12-12-19-04 
Starting the copy process as productiondb-2017-12-12-19-04-us
-----------------------------------------------------------------------------------------------------------------------------
|                                                      CopyDBSnapshot                                                       |
+---------------------------------------------------------------------------------------------------------------------------+
||                                                       DBSnapshot                                                        ||
|+----------------------------+--------------------------------------------------------------------------------------------+|
||  AllocatedStorage          |  800                                                                                       ||
||  DBInstanceIdentifier      |  productiondb                                                                         ||
||  DBSnapshotIdentifier      |  productiondb-2017-12-12-19-04-us                                                     ||
||  Encrypted                 |  False                                                                                     ||
||  Engine                    |  mysql                                                                                     ||
||  EngineVersion             |  5.6.34                                                                                    ||
||  InstanceCreateTime        |  2015-07-10T17:26:16.247Z                                                                  ||
||  Iops                      |  4000                                                                                      ||
||  LicenseModel              |  general-public-license                                                                    ||
||  MasterUsername            |  root                                                                                      ||
||  PercentProgress           |  0                                                                                         ||
||  Port                      |  3306                                                                                      ||
||  SnapshotType              |  manual                                                                                    ||
||  SourceDBSnapshotIdentifier|  arn:aws:rds:ap-southeast-1:52xxxxxxxx33:snapshot:rds:productiondb-2017-12-12-19-04   ||
||  SourceRegion              |  ap-southeast-1                                                                            ||
||  Status                    |  pending                                                                                   ||
||  StorageType               |  io1                                                                                       ||
|+----------------------------+--------------------------------------------------------------------------------------------+|
Looking for DB SnapShot older than 5 days:
 
Following SnapShot is found : productiondb-2017-12-07-19-04-us


Deleting the associated snapshots.... 

-----------------------------------------------------------------------------------------------------------------------------
|                                                     DeleteDBSnapshot                                                      |
+---------------------------------------------------------------------------------------------------------------------------+
||                                                       DBSnapshot                                                        ||
|+----------------------------+--------------------------------------------------------------------------------------------+|
||  AllocatedStorage          |  800                                                                                       ||
||  DBInstanceIdentifier      |  productiondb                                                                         ||
||  DBSnapshotIdentifier      |  productiondb-2017-12-07-19-04-us                                                     ||
||  Encrypted                 |  False                                                                                     ||
||  Engine                    |  mysql                                                                                     ||
||  EngineVersion             |  5.6.34                                                                                    ||
||  InstanceCreateTime        |  2015-07-10T17:26:16.247Z                                                                  ||
||  Iops                      |  4000                                                                                      ||
||  LicenseModel              |  general-public-license                                                                    ||
||  MasterUsername            |  root                                                                                      ||
||  PercentProgress           |  100                                                                                       ||
||  Port                      |  3306                                                                                      ||
||  SnapshotCreateTime        |  2017-12-07T20:45:03.164Z                                                                  ||
||  SnapshotType              |  manual                                                                                    ||
||  SourceDBSnapshotIdentifier|  arn:aws:rds:ap-southeast-1:52xxxxxxxx33:snapshot:rds:productiondb-2017-12-07-19-04   ||
||  SourceRegion              |  ap-southeast-1                                                                            ||
||  Status                    |  deleted                                                                                   ||
||  StorageType               |  io1                                                                                       ||
|+----------------------------+--------------------------------------------------------------------------------------------+|

Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.
Replace productiondb with name of your RDS server.
Add your account id instead of 52xxxxxxx33 in command.
The copy and delete command need to be run from region where it need to be copied and where snapshot is available so region us-east-1 is defined.
Replace in date command to set your own days of which you want to take backup, I have set it to 5 days.
For few initial days when there is no snapshot to delete you will get a error message telling no snapshot found.

Tuesday, June 27, 2017

Deploying application using jenkins build pipeline | Jenkins Tutorial 4 | Certified Jenkins Engineer (CJE)

On deploying application using Jenkins build pileline refer the following video:



Once we have our application tested and working fine than we need them to be deployed to multiple environments like stage than followed by production to get our application and running as soon as possible.

First we need to create a Artifacts for our code.
Go to Post-build Actions and select Archive the artifacts.


Add entry as **/*.war to create war file in job's home directory.


Save and Build the job to create artifacts.
If you see the Console output now you will get the output saying Archiving artifacts finished with Success.


And see the Status of your build where you will get the war file created in job home directory.


Now we need tomcat to be installed and running in our environment to make it all work. Download tomcat from following link.
[root@ip-10-0-1-84 ec2-user]# wget http://apache.mirrors.pair.com/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.zip
--2017-06-27 05:19:09--  http://apache.mirrors.pair.com/tomcat/tomcat-8/v8.5.15/bin/apache-tomcat-8.5.15.zip
Resolving apache.mirrors.pair.com (apache.mirrors.pair.com)... 216.92.2.131
Connecting to apache.mirrors.pair.com (apache.mirrors.pair.com)|216.92.2.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9966102 (9.5M) [application/zip]
Saving to: ‘apache-tomcat-8.5.15.zip’

apache-tomcat-8.5.15.zip                 100%[==================================================================================>]   9.50M  1009KB/s    in 15s     

2017-06-27 05:19:25 (656 KB/s) - ‘apache-tomcat-8.5.15.zip’ saved [9966102/9966102]

[root@ip-10-0-1-84 ec2-user]# unzip apache-tomcat-8.5.15.zip 
[root@ip-10-0-1-84 ec2-user]# cd apache-tomcat-8.5.15
[root@ip-10-0-1-84 apache-tomcat-8.5.15]# ls
bin  conf  lib  LICENSE  logs  NOTICE  RELEASE-NOTES  RUNNING.txt  temp  webapps  work

Give the executable permission to executable files in bin folder.
[root@ip-10-0-1-84 bin]# chmod +x *.sh
[root@ip-10-0-1-84 bin]# ls
bootstrap.jar  catalina-tasks.xml            configtest.bat  digest.bat        setclasspath.sh  startup.bat      tomcat-native.tar.gz  version.bat
catalina.bat   commons-daemon.jar            configtest.sh   digest.sh         shutdown.bat     startup.sh       tool-wrapper.bat      version.sh
catalina.sh    commons-daemon-native.tar.gz  daemon.sh       setclasspath.bat  shutdown.sh      tomcat-juli.jar  tool-wrapper.sh

As I'm running Jenkins on port 8080 so will be running the Tomcat on port  8081 so that there is no conflicts of ports. To do that edit the line no 69 in server.xml file in conf directory with port no 8081.
     69     <Connector port="8081" protocol="HTTP/1.1"

Create a Admin user to run our application by editing the tomcat-user.xml file in conf directory and add similar type of code with username and password as per your need.
  <role rolename="manager-script"/>
  <role rolename="admin-gui"/>
  <user username="tomcat" password="password" roles="manager-script,admin-gui"/>
</tomcat-users>

Now start the Tomcat service to see if all is running fine.
[root@ip-10-0-1-84 conf]# ../bin/startup.sh 
Using CATALINA_BASE:   /home/ec2-user/apache-tomcat-8.5.15
Using CATALINA_HOME:   /home/ec2-user/apache-tomcat-8.5.15
Using CATALINA_TMPDIR: /home/ec2-user/apache-tomcat-8.5.15/temp
Using JRE_HOME:        /usr/lib/jvm/java
Using CLASSPATH:       /home/ec2-user/apache-tomcat-8.5.15/bin/bootstrap.jar:/home/ec2-user/apache-tomcat-8.5.15/bin/tomcat-juli.jar
Tomcat started.

Go to your browser and enter the IP address of your system followed by the port which you have defined for your Tomcat server which is in my case 8081.


Now we need to copy our Artifact in tomcat server so that it can be run and in order to do so we need to install Copy Artifact plugin in Jenkins.
Go to Manage Jenkins than Plugin Manager than Available and search Copy Artifact and install it.


Followed by Deploy to container plugin.


Now Lets create new job which will deploy our war file in tomcat so we can execute it.
I'm creating with name stage-test1, Go to Build and add the project whose war file we want to deploy in my case its git-test1 which I used to test in previous Tutorial. Enter **/*.war to copy all war files from build. Now add Post-build Actions in which we will define our Tomcat credentials and files which need to be passed.


Now we need to define in our original job(git-test1) in which the war file was created to execute tomcat related job(stage-test1) after successful creation on war file.
Go to Configure than Post-build Actions define here files to be copied which are war files here and Project to build name which is stage-test1.


Now Go to Jenkins and Build your initial job which is git-test1 in my case.


You will the second job added in build queue and executed.


You will in the output console of stage-test1 that our war file has been successfully deployed.


Now go to your tomcat server followed by /webapp to see your application running fine.

In the similar way you can add steps to deploy application to your production server as well.

To go to previous Jenkins next tutorial please click following link.

Hope this is being informative for you so please like, share or subscribe to my blog and YouTube Channel.

Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...