Wednesday, December 13, 2017

How to copy RDS snapshot to different region and delete older

Following script can be used to copy the AWS RDS snapshot to different region on daily basis and delete the older than n no of days.

#!/bin/bash
#Script to copy daily DB snapshot to US region and delete older than 5 days
#By Ravi Gadgil

echo -e "----------------------------------\n   `date`   \n----------------------------------"


echo -e "Finding the DB Snapshot ID to be copied."
snapname=$(echo -e "productiondb-`date +%F`-19-04")
newsnapname=$(echo -e "productiondb-`date +%F`-19-04-us")
echo -e "DB Snapshot which is being copied $snapname "

echo -e "Starting the copy process as $newsnapname"
aws rds copy-db-snapshot --source-db-snapshot-identifier arn:aws:rds:ap-southeast-1:52xxxxxxxx33:snapshot:rds:$snapname --target-db-snapshot-identifier $newsnapname --copy-tags --region us-east-1


echo -e "Looking for DB SnapShot older than 5 days:\n "

#Finding SnapShot older than 5 days which needed to be removed
echo "productiondb-`date +%F --date '5 days ago'`-19-04-us" > /tmp/dbsnapdel.txt

if [[ -s /tmp/dbsnapdel.txt ]];
then

echo -e "Following SnapShot is found : `cat /tmp/dbsnapdel.txt`\n"

echo -e "\nDeleting the associated snapshots.... \n"

#Deleting snapshots of Older days
aws rds delete-db-snapshot --db-snapshot-identifier `cat /tmp/dbsnapdel.txt` --region us-east-1

else

echo -e "No SnapShot found older than minimum required no of days"
fi

The output will be similar to this:

----------------------------------
   Tue Dec 12 20:30:02 UTC 2017   
----------------------------------
Finding the DB Snapshot ID to be copied.
DB Snapshot which is being copied productiondb-2017-12-12-19-04 
Starting the copy process as productiondb-2017-12-12-19-04-us
-----------------------------------------------------------------------------------------------------------------------------
|                                                      CopyDBSnapshot                                                       |
+---------------------------------------------------------------------------------------------------------------------------+
||                                                       DBSnapshot                                                        ||
|+----------------------------+--------------------------------------------------------------------------------------------+|
||  AllocatedStorage          |  800                                                                                       ||
||  DBInstanceIdentifier      |  productiondb                                                                         ||
||  DBSnapshotIdentifier      |  productiondb-2017-12-12-19-04-us                                                     ||
||  Encrypted                 |  False                                                                                     ||
||  Engine                    |  mysql                                                                                     ||
||  EngineVersion             |  5.6.34                                                                                    ||
||  InstanceCreateTime        |  2015-07-10T17:26:16.247Z                                                                  ||
||  Iops                      |  4000                                                                                      ||
||  LicenseModel              |  general-public-license                                                                    ||
||  MasterUsername            |  root                                                                                      ||
||  PercentProgress           |  0                                                                                         ||
||  Port                      |  3306                                                                                      ||
||  SnapshotType              |  manual                                                                                    ||
||  SourceDBSnapshotIdentifier|  arn:aws:rds:ap-southeast-1:52xxxxxxxx33:snapshot:rds:productiondb-2017-12-12-19-04   ||
||  SourceRegion              |  ap-southeast-1                                                                            ||
||  Status                    |  pending                                                                                   ||
||  StorageType               |  io1                                                                                       ||
|+----------------------------+--------------------------------------------------------------------------------------------+|
Looking for DB SnapShot older than 5 days:
 
Following SnapShot is found : productiondb-2017-12-07-19-04-us


Deleting the associated snapshots.... 

-----------------------------------------------------------------------------------------------------------------------------
|                                                     DeleteDBSnapshot                                                      |
+---------------------------------------------------------------------------------------------------------------------------+
||                                                       DBSnapshot                                                        ||
|+----------------------------+--------------------------------------------------------------------------------------------+|
||  AllocatedStorage          |  800                                                                                       ||
||  DBInstanceIdentifier      |  productiondb                                                                         ||
||  DBSnapshotIdentifier      |  productiondb-2017-12-07-19-04-us                                                     ||
||  Encrypted                 |  False                                                                                     ||
||  Engine                    |  mysql                                                                                     ||
||  EngineVersion             |  5.6.34                                                                                    ||
||  InstanceCreateTime        |  2015-07-10T17:26:16.247Z                                                                  ||
||  Iops                      |  4000                                                                                      ||
||  LicenseModel              |  general-public-license                                                                    ||
||  MasterUsername            |  root                                                                                      ||
||  PercentProgress           |  100                                                                                       ||
||  Port                      |  3306                                                                                      ||
||  SnapshotCreateTime        |  2017-12-07T20:45:03.164Z                                                                  ||
||  SnapshotType              |  manual                                                                                    ||
||  SourceDBSnapshotIdentifier|  arn:aws:rds:ap-southeast-1:52xxxxxxxx33:snapshot:rds:productiondb-2017-12-07-19-04   ||
||  SourceRegion              |  ap-southeast-1                                                                            ||
||  Status                    |  deleted                                                                                   ||
||  StorageType               |  io1                                                                                       ||
|+----------------------------+--------------------------------------------------------------------------------------------+|

Note : You need to have AWS CLI installed in you server to get this script running fine if in case you don't have that follow the following link.
Replace productiondb with name of your RDS server.
Add your account id instead of 52xxxxxxx33 in command.
The copy and delete command need to be run from region where it need to be copied and where snapshot is available so region us-east-1 is defined.
Replace in date command to set your own days of which you want to take backup, I have set it to 5 days.
For few initial days when there is no snapshot to delete you will get a error message telling no snapshot found.

Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...