Wednesday, August 26, 2015

Sed command to replace wild card and special characters.

Sed is a very powerful tool to replace characters in Linux. It is very helpful in scripts when we need to replace entries with new one or create a whole new entry.

To do simple character replace, here mp4 is replaced by abc:
[root@ip-10-0-1-24 ravi]# cat test.txt 
new= -Days-Shipping-With-Swift-and-VIPER.mp4
new= -Ways-to-Enrich-the-Tech-Industry.mp4
new= All-the-IO-News-That-You-Should-Care-About.mp4
new= BottomUp-Programming-in-Swift.mp4

[root@ip-10-0-1-24 ravi]# sed -i 's/mp4/abc/g' test.txt 

[root@ip-10-0-1-24 ravi]# cat test.txt 
new= -Days-Shipping-With-Swift-and-VIPER.abc
new= -Ways-to-Enrich-the-Tech-Industry.abc
new= All-the-IO-News-That-You-Should-Care-About.abc
new= BottomUp-Programming-in-Swift.abc

General syntax : sed -i 's/word to be replaced/new word/g' filename

To replace characters with wildcard, here numericals are replaced new:
[root@ip-10-0-1-24 ravi]# cat test3.txt 
FILENAME1
FILENAME2
FILENAME3
FILENAME4

[root@ip-10-0-1-24 ravi]# sed -i 's/FILENAME.*/new/g' test3.txt 

[root@ip-10-0-1-24 ravi]# cat test3.txt 
new
new
new
new

To replace with wildcard "." followed by * need to be used.

To replace special characters, here special character are replaced with new:
[root@ip-10-0-1-24 ravi]# cat test.txt 
FILENAME[0]= -Days-Shipping-With-Swift-and-VIPER.mp4
FILENAME[1]= -Ways-to-Enrich-the-Tech-Industry.mp4
FILENAME[2]= All-the-IO-News-That-You-Should-Care-About.mp4
FILENAME[3]= BottomUp-Programming-in-Swift.mp4

[root@ip-10-0-1-24 ravi]#  sed -i 's/FILENAME\[.*\]/new/g' test.txt 

[root@ip-10-0-1-24 ravi]# cat test.txt 
new= -Days-Shipping-With-Swift-and-VIPER.mp4
new= -Ways-to-Enrich-the-Tech-Industry.mp4
new= All-the-IO-News-That-You-Should-Care-About.mp4
new= BottomUp-Programming-in-Swift.mp4

To replace with special character, special character need to be mentioned with "\" followed by character.





Command to sort Domain Names with respect to there TTL.

If in case you want to sort the Domain Names in order of TTL then following Commands can be used.

First create a file listing all the Domains which need to be checked.
[root@ip-10-0-1-24 ravi]# cat test.txt 
ib.theteamie.com
uwc.teamieapp.com
nbc.com

Following command will give the domains in sorted order.
[root@ip-10-0-1-24 ravi]# for i in `cat test.txt` ; do dig +noauthority +noquestion +nostats $i @8.8.8.8 | grep -A1 SECTION |tail -n 1 | awk '{print $2" " $1 }' | sort -n ; done
299 ib.theteamie.com.
293 uwc.teamieapp.com.
9 nbc.com.

dig : command to show DNS record.
+noauthority, +noquestion, +nostats : To get TTL only.
@8.8.8.8 : To get without cached TTL.
grep -A1 : To get next line after grepped item.
 

Monday, August 24, 2015

How to create users from ansible with public key and password.

Ansible is used for centrally managing the tasks and one of the major task is user management. So in order to perform this either we will be creating the users with passwords or by there public keys which is one of the preferred way as well.

To add user with password first we need to create a encrypted password from command line which we can forward to our Ansible playbook.
 
[root@localhost Desktop]# python -c 'import crypt; print crypt.crypt("userpassword", "user")'
usx7b002w0mBw

Use the "usx7b002w0mBw" password in your Ansible playbook to set user's password. The required Ansible playbook will be like this.

---
- hosts: stage
  remote_user: ec2-user
  sudo: yes
  tasks:
      - name: Add User 
        user: name=user
              groups=wheel,dev-team
       password=usx7b002w0mBw

hosts : To servers whom you want to add user.
remote_user : From which user you want to run your commands.
sudo : Run commands as sudo.
name : Name of user to be created.
groups : To which group user need to be added.
password : password of the user.

To add user with public key instead of password use the following playbook.

---
- hosts: stage
  remote_user: ec2-user
  sudo: yes
  tasks:
      - name: Add User 
        user: name=user1
              groups=wheel,dev-team

      - name: Adding keys to authorized_keys
        authorized_key: user=user1 key="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDfrz+CkYp6k7O/g8+K9RHmXzCAJaRHW4xrpme+l5O6E3LBLi+BcmDg/MV2RI8OU69jxtZ2O6jdIVHts1O1ZyAEznBJos2NgeGPIcgrp5ZVax+yBa92TeXxJlkEEMlT1RGkgy3GjSqobhsqsLBHmDiHI5921f9mECIO3kUgqeMTXw== localhost@localserver" path=/home/user1/.ssh/authorized_keys state=present
        with_items: ssh_keys.results

authorized_key : Place your public key here and set it for user1.

In order to provide sudo access to created user we need to enable sudo group in /etc/sudoers file

## Allows people in group wheel to run all commands
# %wheel        ALL=(ALL)       ALL

After enabling this group add the new users to this group so they can have sudo access.

Wednesday, August 19, 2015

Script to take daily backup of S3 buckets.

For a total Disaster recovery environment we should have backup of S3 buckets as well if in case it got deleted by accident. I am taking the backup of all S3 buckets in S3 it self by using the AWS CLI tool.

Following script can be help full in this case:
#!/bin/bash
# Created by Ravi Gadgil
# Script to daily sync data of S3 buckets in S3 bucket

echo -e "\n-----------------------------\n Starting the sync for `date`..... \n------------------------------\n"

aws s3 ls | awk '{print $3}' | grep -v 'S3-daily-sync\|production-crocodoc' > /tmp/s3buckets.txt

for i in `cat /tmp/s3buckets.txt` ; do aws s3 sync s3://$i s3://S3-daily-sync/$i/ ; done


aws s3 ls : Is used to get name of all the buckets which are in your S3.
grep -v : Is used to exclude specific buckets of which you don't want to sync. You should always put the bucket to exclude which your using to take rest of S3 buckets backup as in my case its "S3-daily-sync" else it will keep syncing it self in loop.

Put this in cron when you want this daily backup to happen.

How to upgrade Ansible to latest version.

Ansible releases it update in Git very frequently and to get the most of the updated modules and package its recommended to keep it updated.

You need to go the directory where you have installed the Ansible in my case its /root/ansible.
Following is the way to update Ansible:
[root@server downloads]# cd /root/ansible/
[root@server ansible]# git pull --rebase
[root@server ansible]# git submodule update --init --recursive

If you have used EC2 module to access AWS resources you should also update the files in /etc/ansible as well. To know how to use Ansible with AWS you can use following link.

Following is the way to update:
[root@server ansible]# cp contrib/inventory/ec2.py /etc/ansible/hosts
[root@server ansible]# cp contrib/inventory/ec2.ini /etc/ansible/ec2.ini
[root@server ansible]# cp examples/ansible.cfg /etc/ansible/ansible.cfg

Note: After updating ansible.cfg do make the required changes which you have made in your previous ansible.cfg.


Friday, August 14, 2015

How to install and configure Ansible for AWS in EC2 Linux.

Ansible is a very good open source configuration management and automation tool which can run on any machine which has SSH and Python working on it. There is no need of client server architecture and any other language.
It has pre built commands in it as well and we can write out own with YML language. It can run any type of scripting language.

To install Ansible use following steps:
[root@server downloads]# cd ~
[root@server root]# git clone git://github.com/ansible/ansible.git --recursive
[root@server root]# cd ansible/
[root@server ansible]# source ./hacking/env-setup

env-setup is used to set the environment variables for Ansible so to make it permanent its recommended to add it in .bashrc.
[root@server root]# echo "source ~/ansible/hacking/env-setup" >> ~/.bashrc


There are few dependencies for Ansible which need to be install.
[root@server root]# easy_install pip[root@server root]# pip install paramiko PyYAML Jinja2 httplib2 six
[root@server root]# ansible --version
ansible 1.9.1

Following steps will help us to setup Ansible configuration file as well as the hosts file according to which it will take the server names of our enviornment.
[root@server root]# mkdir /etc/ansible
[root@server root]# cp ~/ansible/contrib/inventory/ec2.* /etc/ansible/
[root@server root]# cp ~/ansible/examples/ansible.cfg /etc/ansible/
[root@server root]# cd /etc/ansible/[root@server ansible]# mv ec2.py hosts

Now to make it interact with your AWS environment either provide the AWS IMA to authenticate the user by defining a role at the time of server boot or we can create a .boto file your home directory from where Ansible can use our credentials.
[root@server root]# # cat ~/.boto 
[Credentials]
aws_access_key_id = AXXXXXXXXXXXXXXXXXXXXXQ
aws_secret_access_key = 2XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXT

Now we are good to communicate with our AWS environment, run the hosts script to get the all servers of AWS tagged according to there types.
[root@server ansible]#  ./hosts 
{
    "tag_Name_ansible": [
    "ec2-54-XX-XX-149.ap-southeast-1.compute.amazonaws.com"
  ],
  "type_t1_micro": [
    "ec2-46-XX-XX-239.ap-southeast-1.compute.amazonaws.com",
    "ec2-46-XX-XX-177.ap-southeast-1.compute.amazonaws.com",
    "ec2-54-XX-XX-149.ap-southeast-1.compute.amazonaws.com"
  ]
}

Ansible will list all the servers with all the possible types it can be sorted and then we can use this to run commands on it.

We can use password of user from which we want to execute the commands but that's not a recommended way so either we make our servers password less to each other or use the ssh keys to pass in the authentication.

I prefer the passing the ssh keys to do the authentication so following steps can help in that.
[root@server root]# eval `ssh-agent -s` 
Agent pid 2173
[root@server root]# ssh-add /root/.ssh/Ravi-test1.pem
Identity added: /root/.ssh/Ravi-test1.pem (/root/.ssh/Ravi-test1.pem)

To add the key we need to start the ssh-agent and ssh-add to add the pem file.

Now we can run Ansible commands to see how its working, Ravi-test1.pem is the key for server which I am using for Ansible it self so will try to ping it and see how it goes.
[root@server root]# ansible -m ping -u ec2-user tag_Name_ansible
ec2-54-XX-XX-149.ap-southeast-1.compute.amazonaws.com | success >> {
    "changed": false,
    "ping": "pong"
}

If we need to run any specific command on servers of any specific tag we can use the following command, I am using -u to define user, -s to run as sudo and -a to execute a specific command.
[root@server root]# ansible tag_Name_ansible -u ec2-user -s -a "id"
ec2-54-179-72-149.ap-southeast-1.compute.amazonaws.com | success | rc=0 >>
uid=0(root) gid=0(root) groups=0(root)

To stop SSH key checking every time do the following entry in ansible.cfg
# uncomment this to disable SSH key host checking
host_key_checking = False

Wednesday, August 12, 2015

How to manage DNS record via AWS Route 53

AWS offers a get product as Route 53 which can be be used to manage our DNS records. Great thing about Route 53 is that is can be managed by AWS CLI which helps in automating things via scripts.
Following commands can be used to do day today work on Route 53 via command line..

To check how many Hosted Zones are present.
[root@server newsite-setup]# aws route53 list-hosted-zones
----------------------------------------------------------------------
|                           ListHostedZones                          |
+--------------------------------------------------------------------+
||                            HostedZones                           ||
|+-------------------------+----------------------------------------+|
||  CallerReference        |  1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd  ||
||  Id                     |  /hostedzone/ZXXXXXXXXXXXX6            ||
||  Name                   |  testsite.com.                         ||
||  ResourceRecordSetCount |  212                                   ||
|+-------------------------+----------------------------------------+|
||                            HostedZones                           ||
|+-------------------------+----------------------------------------+|
||  CallerReference        |  7xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxE  ||
||  Id                     |  /hostedzone/ZXXXXXXXXXXXXL            ||
||  Name                   |  testsite.asia.                        ||
||  ResourceRecordSetCount |  4                                     ||
|+-------------------------+----------------------------------------+|
|||                             Config                             |||
||+--------------------+-------------------------------------------+||
|||  Comment           |  Test Site                                |||
||+--------------------+-------------------------------------------+||


To get the name servers of the DNS hosted zone:
[root@server newsite-setup]# aws route53 get-hosted-zone --id ZXXXXXXXXXXXX
----------------------------------------------------------------------
|                            GetHostedZone                           |
+--------------------------------------------------------------------+
||                           DelegationSet                          ||
|+------------------------------------------------------------------+|
|||                           NameServers                          |||
||+----------------------------------------------------------------+||
|||  ns-1162.awsdns-17.org                                         |||
|||  ns-1621.awsdns-10.co.uk                                       |||
|||  ns-907.awsdns-49.net                                          |||
|||  ns-293.awsdns-36.com                                          |||
||+----------------------------------------------------------------+||
||                            HostedZone                            ||
|+-------------------------+----------------------------------------+|
||  CallerReference        |  7xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx2  ||
||  Id                     |  /hostedzone/ZXXXXXXXXXXXXX            ||
||  Name                   |  newsite.org.                          ||
||  ResourceRecordSetCount |  3                                     ||
|+-------------------------+----------------------------------------+|
|||                             Config                             |||
||+---------------------------+------------------------------------+||
|||  Comment                  |  New Site                          |||
||+---------------------------+------------------------------------+||

To get list of all DNS records associated with a hosted zone:
[root@server newsite-setup]# aws route53 list-resource-record-sets --hosted-zone-id ZXXXXXXXXXXXX
--------------------------------------------------------------------------------------
|                               ListResourceRecordSets                               |
+------------------------------------------------------------------------------------+
||                                ResourceRecordSets                                ||
|+----------------------------------------+----------------------+------------------+|
||                  Name                  |         TTL          |      Type        ||
|+----------------------------------------+----------------------+------------------+|
||  newsite  .org.                        |  172800              |  NS              ||
|+----------------------------------------+----------------------+------------------+|
|||                                 ResourceRecords                                |||
||+--------------------------------------------------------------------------------+||
|||                                      Value                                     |||
||+--------------------------------------------------------------------------------+||
|||  ns-293.awsdns-36.com.                                                         |||
|||  ns-1162.awsdns-17.org.                                                        |||
|||  ns-1621.awsdns-10.co.uk.                                                      |||
|||  ns-907.awsdns-49.net.                                                         |||
||+--------------------------------------------------------------------------------+||
||                                ResourceRecordSets                                ||
|+--------------------------------------------+-----------------+-------------------+|
||                    Name                    |       TTL       |       Type        ||
|+--------------------------------------------+-----------------+-------------------+|
||  newsite.org.                              |  900            |  SOA              ||
|+--------------------------------------------+-----------------+-------------------+|
|||                                 ResourceRecords                                |||
||+--------------------------------------------------------------------------------+||
|||                                      Value                                     |||
||+--------------------------------------------------------------------------------+||
|||  ns-293.awsdns-36.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400  |||
||+--------------------------------------------------------------------------------+||
||                                ResourceRecordSets                                ||
|+----------------------------------------------------+-------------+---------------+|
||                        Name                        |     TTL     |     Type      ||
|+----------------------------------------------------+-------------+---------------+|
||  test.newsite.org.                                 |  300        |  A            ||
|+----------------------------------------------------+-------------+---------------+|
|||                                 ResourceRecords                                |||
||+--------------------------------------------------------------------------------+||
|||                                      Value                                     |||
||+--------------------------------------------------------------------------------+||
|||  1.2.3.4                                                                       |||
||+--------------------------------------------------------------------------------+||


To check the DNS record of a specific DNS record under a hosted zone:
[root@server newsite-setup]# aws route53 list-resource-record-sets --hosted-zone-id ZXXXXXXXXXXXX6 --start-record-name demo.newsite.com --max-items 1
------------------------------------------------------------------------------
|                           ListResourceRecordSets                           |
+--------------+------------+-----------------------------+------------------+
|  IsTruncated | MaxItems   |       NextRecordName        | NextRecordType   |
+--------------+------------+-----------------------------+------------------+
|  True        |  1         |  new.demo.newsite.com.      |  CNAME           |
+--------------+------------+-----------------------------+------------------+
||                            ResourceRecordSets                            ||
|+-----------------------------------------------------+--------------------+|
||                        Name                         |       Type         ||
|+-----------------------------------------------------+--------------------+|
||  demo.newsite.com.                                  |  CNAME             ||
|+-----------------------------------------------------+--------------------+|
|||                               AliasTarget                              |||
||+----------------------------+------------------------+------------------+||
|||           DNSName          | EvaluateTargetHealth   |  HostedZoneId    |||
||+----------------------------+------------------------+------------------+||
|||  auto.newsite.com.         |  False                 |  ZXXXXXXXXXXXX6  |||
||+----------------------------+------------------------+------------------+||

To create a DNS new entry we need to create a json file with what details we want to add and after that need to pass that to Route 53 via CLI.

Following is a example of a simple A record json file:
{
  "Comment": "A new record set for the zone.",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "test.newsite.com.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "1.2.3.4"
          }
        ]
      }
    }
  ]
}

For a simple CNAME record:
{
  "Comment": "A new record set for the zone.",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "test2.newsite.com.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "demo.newsite.com."
          }
        ]
      }
    }
  ]
}

For DNS record Alias:
{
  "Comment": "A new record set for the zone.",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "test2.newsite.com.",
        "Type": "CNAME",
        "AliasTarget": {
          "HostedZoneId": "ZXXXXXXXXXXXXXA",
          "DNSName": "demo.newsite.com.",
          "EvaluateTargetHealth": false
        }
      }
    }
  ]
}

To delete any DNS record replace the "CREATE" in json file with "DELETE" and rest will be same.

To execute this json file following command will be used with path to json file.
[root@server tmp]# aws --debug route53 change-resource-record-sets --hosted-zone-id ZXXXXXXXXXXXXA --change-batch file:///tmp/test.json
-----------------------------------------------------
|             ChangeResourceRecordSets              |
+---------------------------------------------------+
||                   ChangeInfo                    ||
|+--------------+----------------------------------+|
||  Comment     |  A new record set for the zone.  ||
||  Id          |  /change/CXXXXXXXXXXXX5          ||
||  Status      |  PENDING                         ||
||  SubmittedAt |  2015-08-12T10:42:37.348Z        ||
|+--------------+----------------------------------+|

To know the status of your DNS record:
[root@teamie-auto ~]# aws route53 get-change --id CXXXXXXXXXXF
----------------------------------------------------------------------------------------------------
|                                             GetChange                                            |
+--------------------------------------------------------------------------------------------------+
||                                           ChangeInfo                                           ||
|+--------------------------------+------------------------+---------+----------------------------+|
||             Comment            |          Id            | Status  |        SubmittedAt         ||
|+--------------------------------+------------------------+---------+----------------------------+|
||  A new record set for the zone.|  /change/CXXXXXXXXXXF  |  INSYNC |  2015-07-16T07:51:28.614Z  ||
|+--------------------------------+------------------------+---------+----------------------------+|



Manage AWS Load Balancer Certificates.

AWS EC2 Load Balancer certificates can be further manage by AWS CLI.

To get list of Certificates available:

[root@server ec2-user]# aws iam list-server-certificates
----------------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                    ListServerCertificates                                                                    |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
||                                                                ServerCertificateMetadataList                                                               ||
|+-----------------------------------------------------------------------+-------+------------------------+--------------------------+------------------------+|
||                                  Arn                                  | Path  |  ServerCertificateId   |  ServerCertificateName   |      UploadDate        ||
|+-----------------------------------------------------------------------+-------+------------------------+--------------------------+------------------------+|
||  arn:aws:iam::5xxxxxxxxxx3:server-certificate/Cert1                   |  /    |  AXXXXXXXXXXXXXXXXXXXO |  Cert1                   |  2013-04-13T12:45:04Z  ||
||  arn:aws:iam::5xxxxxxxxxx3:server-certificate/Cert2                   |  /    |  AXXXXXXXXXXXXXXXXXXXM |  Cert2                   |  2014-05-20T05:12:25Z  ||
|+-----------------------------------------------------------------------+-------+------------------------+--------------------------+------------------------+|

To delete a specific Certificate use following command:

[root@server downloads]# aws iam delete-server-certificate --server-certificate-name Cert1

To get all information about any certificate including SSL certificates uploaded:

[root@server downloads]# aws iam get-server-certificate --server-certificate-name Cert1


 

Set rule in S3 to give all access to files added in specific folder.

In S3 we can't set permissions on specific folder with in the S3 bucket and sometimes we need to set a global rule for files under such specific folder. We can use Bucket Policy in this case to do this for us.

Following is the rule to give global access to files all uploaded in /global-data folder so it can be downloaded.

To set this up go to root location of your bucket than permissions and select edit bucket policy.

{
 "Version": "2012-10-17",
 "Id": "Policy1438583545455",
 "Statement": [
  {
   "Sid": "Stmt1438583521051",
   "Effect": "Allow",
   "Principal": "*",
   "Action": "s3:*",
   "Resource": "arn:aws:s3:::test-bucket/global-data/*"
  }
 ]
}

How to create a super user in MySQL RDS.

RDS by default has a root user but if we need to create a user with all access or limited access depending upon need following commands are helpful.

mysql> CREATE USER 'superuser'@'%' IDENTIFIED BY 'Su93RU53R';
Query OK, 0 rows affected (0.04 sec)

mysql> GRANT ALL ON `%`.* TO superuser@`%`
Query OK, 0 rows affected (0.01 sec)

To create a user with only read access.

mysql> CREATE USER 'dummy'@'%' IDENTIFIED BY 'Dummy123@';
Query OK, 0 rows affected (0.04 sec)

mysql> GRANT SELECT ON `%`.* TO dummy@`%`;
Query OK, 0 rows affected (0.01 sec)

Grant SELECT will let user to view every thing on DB but not able to edit any thing.

To check user access of any user following command can be used.

mysql> show grants for dummy ;
+------------------------------------------------------------------------------------------------------+
| Grants for dummy@%                                                                                   |
+------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'dummy'@'%' IDENTIFIED BY PASSWORD '*8EC0AF9740D5D97644ACA2D6CB8A4B7A8F7319D4' |
| GRANT SELECT ON `%`.* TO 'dummy'@'%'                                                                 |
+------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

To update the user password following command can be used.

SET PASSWORD FOR 'dummy'@'%' = PASSWORD ('ChangeWh3nNeeded');



Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...