How to install python2.7 with pip2.7

Python is one of the most famous and powerful languages used so in order to install or update it to 2.7 version following steps can be used.

Installing the Python2.7.

[root@ip-10-0-1-55 ~]# yum install python27 python27-devel
[root@ip-10-0-1-55 ~]# python --version
Python 2.7.10

If its still showing older version add the new version as default.

[root@ip-10-0-1-55 ~]# alternatives --config python

There are 2 programs which provide 'python'.
 + 1           /usr/bin/python2.6
*  2           /usr/bin/python2.7

Enter to keep the current selection[+], or type selection number: 2
[root@ip-10-0-1-55 ~]# python --version
Python 2.7.10

Installing the pip2.7.

[root@ip-10-0-1-55 ~]# wget https://bootstrap.pypa.io/get-pip.py
--2016-05-26 11:10:33--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 103.245.222.175
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|103.245.222.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1524722 (1.5M) [text/x-python]
Saving to: ‘get-pip.py’

100%[=====================================================================>] 1,524,722   --.-K/s   in 0.09s   

2016-05-26 11:10:33 (16.2 MB/s) - ‘get-pip.py’ saved [1524722/1524722]
[root@ip-10-0-1-55 ~]# python2.7 get-pip.py
Collecting pip
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 675kB/s 
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 7.1MB/s 
Installing collected packages: pip, wheel
Successfully installed pip-8.1.2 wheel-0.29.0

[root@ip-10-0-1-55 dist-packages]# which pip2.7
/usr/bin/pip2.7

Installing dependencies or packages from pip2.7.

[root@ip-10-0-1-55 ~]# pip2.7 install asciitree
Collecting asciitree
  Downloading asciitree-0.3.1.tar.gz
Building wheels for collected packages: asciitree
  Running setup.py bdist_wheel for asciitree ... done
  Stored in directory: /root/.cache/pip/wheels/10/14/df/88531e6ba23be75397d69921e580464f2850b6f45e982ac644
Successfully built asciitree
Installing collected packages: asciitree
Successfully installed asciitree-0.3.1

How to create and remove swap partition in Linux

If our server is having memory issues and we want to increase it without updating the physical RAM than Swap is a very good option. It's slower than the physical RAM but can do the tasks also if created on faster hard disks the results are good.

Create a Swap partition in place where you wanna create it. I am creating it in /swap with 2GB of size.

[root@ip-10-0-1-38 /]# dd if=/dev/zero of=/swap bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 54.8004 s, 39.2 MB/s

Update the permissions of /swap partition to 600.

[root@ip-10-0-1-38 /]# swapon /swap 
swapon: /swap: insecure permissions 0644, 0600 suggested.
[root@ip-10-0-1-38 /]# chmod 600 /swap 

Start the Swap on /swap partition.

[root@ip-10-0-1-38 /]# swapon /swap 

To check if swap has started or not.

[root@ip-10-0-1-38 /]# swapon -s
Filename                                Type            Size    Used    Priority
/swap                                   file    2097148 0       -1

[root@ip-10-0-1-38 /]# free -m
             total       used       free     shared    buffers     cached
Mem:          3752       2353       1399          0         39       2184
-/+ buffers/cache:        128       3624
Swap:         2047          0       2047

Add the entry in /etc/fstab so that swap can start on server boot.

/swap swap swap defaults 0 0

If in case you face following error stop and start of swap can solve the issue.

[root@ip-10-0-1-38 /]# swapon /swap 
swapon: /swap: swapon failed: Device or resource busy

[root@ip-10-0-1-38 /]# swapoff /swap 
[root@ip-10-0-1-38 /]# swapon /swap
[root@ip-10-0-1-38 /]# free -m
             total       used       free     shared    buffers     cached
Mem:          3752       2353       1399          0         39       2184
-/+ buffers/cache:        128       3624
Swap:         2047          0       2047

In order to remove swap partition following steps can be used.

[root@ip-10-0-1-38 ec2-user]# swapoff /swap
[root@ip-10-0-1-38 ec2-user]# rm -rf /swap 

How to install SyntaxNet in Linux

SyntaxNet is an open-source neural network framework for TensorFlow that provides a foundation for Natural Language Understanding (NLU) systems.
In order to install it in Linux servers following steps can be used..

It requires Python 2.7 so if you don't have that install the same.

[root@ip-10-0-1-55 ~]# yum install python27 python27-devel
[root@ip-10-0-1-55 ~]# python --version
Python 2.7.10

If its still showing older version add the new version as default.

[root@ip-10-0-1-55 ~]# alternatives --config python

There are 2 programs which provide 'python'.
 + 1           /usr/bin/python2.6
*  2           /usr/bin/python2.7

Enter to keep the current selection[+], or type selection number: 2
[root@ip-10-0-1-55 ~]# python --version
Python 2.7.10

Install the java1.8 version.

[root@ip-10-0-1-55 ~]# yum install java-1.8.0-openjdk*

If old java version is showing up update it via following command.

[root@ip-10-0-1-55 ~]# alternatives --config java

Make sure that your java home is pointing to java directory where javac is placed else it will show the java not found error. In my case it was placed in /usr/lib/jvm/java-1.8.0-openjdk.x86_64/ so updated it to that location.

[root@ip-10-0-1-55 ~]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64/[root@ip-10-0-1-55 ~]# echo $JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk.x86_64/
[root@ip-10-0-1-55 ~]# java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Installing the bazel. As I am installing the Linux version so taking the executable from here.

[root@ip-10-0-1-55 ~]# wget https://github.com/bazelbuild/bazel/releases/download/0.2.2b/bazel-0.2.2b-installer-linux-x86_64.sh
[root@ip-10-0-1-55 ~]# chmod +x bazel-0.2.2b-installer-linux-x86_64.sh [root@ip-10-0-1-55 ~]# ./bazel-0.2.2b-installer-linux-x86_64.sh 

Installing the swig package.

[root@ip-10-0-1-55 ~]# yum install swig

If your using the python2.7 than you need to install python packages using the pip2.7 as well so first we need to add that. If your not using python2.7 normal pip install will work fine.

[root@ip-10-0-1-55 ~]# wget https://bootstrap.pypa.io/get-pip.py
--2016-05-26 11:10:33--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 103.245.222.175
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|103.245.222.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1524722 (1.5M) [text/x-python]
Saving to: ‘get-pip.py’

100%[=====================================================================>] 1,524,722   --.-K/s   in 0.09s   

2016-05-26 11:10:33 (16.2 MB/s) - ‘get-pip.py’ saved [1524722/1524722]
[root@ip-10-0-1-55 ~]# python2.7 get-pip.py
Collecting pip
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 675kB/s 
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 7.1MB/s 
Installing collected packages: pip, wheel
Successfully installed pip-8.1.2 wheel-0.29.0

[root@ip-10-0-1-55 dist-packages]# which pip2.7
/usr/bin/pip2.7

Install the required packages protobuf, asciitree, numpy.

[root@ip-10-0-1-55 ~]# pip2.7 install -U protobuf==3.0.0b2
Collecting protobuf==3.0.0b2
  Downloading protobuf-3.0.0b2-py2.py3-none-any.whl (326kB)
    100% |████████████████████████████████| 327kB 2.2MB/s 
Collecting six>=1.9 (from protobuf==3.0.0b2)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting setuptools (from protobuf==3.0.0b2)
  Downloading setuptools-21.2.1-py2.py3-none-any.whl (509kB)
    100% |████████████████████████████████| 512kB 1.6MB/s 
Installing collected packages: six, setuptools, protobuf
  Found existing installation: six 1.8.0
    Uninstalling six-1.8.0:
      Successfully uninstalled six-1.8.0
  Found existing installation: setuptools 12.2
    Uninstalling setuptools-12.2:
      Successfully uninstalled setuptools-12.2
Successfully installed protobuf-3.0.0b2 setuptools-21.2.1 six-1.10.0
[root@ip-10-0-1-55 ~]# pip2.7 install asciitree
Collecting asciitree
  Downloading asciitree-0.3.1.tar.gz
Building wheels for collected packages: asciitree
  Running setup.py bdist_wheel for asciitree ... done
  Stored in directory: /root/.cache/pip/wheels/10/14/df/88531e6ba23be75397d69921e580464f2850b6f45e982ac644
Successfully built asciitree
Installing collected packages: asciitree
Successfully installed asciitree-0.3.1
[root@ip-10-0-1-55 ~]# pip2.7 install numpy
Collecting numpy
  Downloading numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
    100% |████████████████████████████████| 15.3MB 48kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.11.0
[root@ip-10-0-1-55 ~]# pip freeze | grep protobuf
protobuf==3.0.0b2
[root@ip-10-0-1-55 ~]# pip freeze | grep numpy
numpy==1.11.0
[root@ip-10-0-1-55 ~]# pip freeze | grep asciitree
asciitree==0.3.1

Building the SyntaxNet from Git.

[root@ip-10-0-1-55 ~]# git clone --recursive https://github.com/tensorflow/models.git 
[root@ip-10-0-1-55 ~]# cd models/syntaxnet/tensorflow 
[root@ip-10-0-1-55 ~]# ./configure 
[root@ip-10-0-1-55 ~]# cd .. 
[root@ip-10-0-1-55 ~]# bazel test syntaxnet/... util/utf8/...

Testing the installation.

[root@ip-10-0-1-55 ~]# echo 'Ravi' | syntaxnet/demo.sh

Take dump of all the databases from mysql server

To take MySQL dump of all the databases following script can be used it will work for normal MySQL server as well as Amazon RDS.

[root@ip-10-0-1-231 ravi]# cat dump.sh
#!/bin/bash
#Script to get dump of all the databases within the server.

USER="root"
PASSWORD="dbpassword"

databases=`mysql -h prod-XXXXXXXXXXXX.rds.amazonaws.com -u $USER -p$PASSWORD -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`

for db in $databases; do
    if [[ "$db" != "information_schema" ]] && [[ "$db" != "performance_schema" ]] && [[ "$db" != "mysql" ]] && [[ "$db" != _* ]] ; then
        echo "Dumping database: $db"
        mysqldump -h prod-XXXXXXXXXXXXXX.rds.amazonaws.com -u $USER -p$PASSWORD --databases $db > `date +%Y%m%d`.$db.sql
       # gzip $OUTPUT/`date +%Y%m%d`.$db.sql
    fi
done

How to add numbers via bash

There are cases when we need to add up numbers or the outputs of our commands via bash so in that case following commands are helpful.

[ec2-user@ip-10-0-1-38 ~]$ cat list
45
78
56
67
34
56

To do the sum:

[ec2-user@ip-10-0-1-38 ~]$ cat list | awk '{ SUM += $1} END { print SUM }'
336

Same can be used to get sum of any output for example grep.

[ec2-user@ip-10-0-1-38 ~]$ cat s3data.txt | grep Size | grep  Mi | awk '{ print $3 }' | awk '{ SUM += $1} END { print SUM }'
3405

Find size of S3 Buckets

In order to find the size of S3 buckets we can use following ways:

First Method: via s3api cli

[root@ip-10-0-1-231 ravi]# aws s3api list-objects --bucket bucketname --output json --query "[sum(Contents[].Size), length(Contents[])]"
[
    30864102, 
    608
]

30864102: Is the size in Bytes.
608: No of objects in bucket.

Second Method : via s3 cli

[root@ip-10-0-1-231 ravi]# aws s3 ls s3://bucketname --recursive  | grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}'
29.4343 MB

Using bash commands to get output in desired way.

Third Method : via s3 cli with parameters

[root@ip-10-0-1-231 ravi]# aws s3 ls s3://bucketname --recursive --human-readable --summarize
2016-05-04 11:32:00    7.6 KiB prompthooks0.py

Total Objects: 1
   Total Size: 7.6 KiB

-human-readable: this will provide the data that is already in KB, MB, GB. TB etc.

To get size of all the buckets in your S3 use the following script:

root@ip-10-0-1-231 ravi]# cat s3size.sh 
#!/bin/bash
#Script to find the size of all S3 buckets.

aws s3 ls | awk '{ print $3 }' > /home/ec2-user/s3.list
for i in `cat /home/ec2-user/s3.list` ; do
echo -e "\nBucket Name: $i"
aws s3 ls s3://$i --recursive --human-readable --summarize | grep Total ; done

output of script:

root@ip-10-0-1-231 ravi]# tail s3data.txt
Bucket Name: bucket1
Total Objects: 17676
   Total Size: 67.3 GiB

Bucket Name: bucket2
Total Objects: 1
   Total Size: 7.6 KiB

Bucket Name: bucket3
Total Objects: 5302
   Total Size: 3.5 GiB