DevOps / Sys Admin Q & A #1B : Linux Commands

egrep is equivalent to grep -E:

$ grep -E '^no(fork|group)' /etc/group
nogroup:x:65534:

$ egrep '^no(fork|group)' /etc/group
nogroup:x:65534:

The "-E" switches grep into a special mode so that the expression is evaluated as an extended regular expression as opposed to its normal pattern matching. So, in the example above, it looked for a pattern starting "nofork" or "nogroup".

We can count the number of files in a directory using the egrep -c:

$ ls -la |egrep -c '^-'
58

As we aleady know, we can also count the number of lines using "wc" instead of using the count flag, "c":

$ ls -l |egrep '^-' | wc -l
58

The command dd is used to convert and copy files.

By default, dd reads from stdin and writes to stdout, but these can be changed by using the if (input file) and of (output file) options.

Backing up a disk to an image will be faster than copying the exact data. Also, disk image make the restoration much more easier. We can create a compressed disk image using dd:

$ sudo dd if=/dev/sda1 | gzip >/tmp/sda1disk.img.gz

We often use dd to simulate CPU load by filling a file with random content:

$ dd if=/dev/urandom of=500MBfile bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 103.075 s, 5.1 MB/s

For this, /dev/urandom will supply random numbers, which will be generated by the kernel. This will lead to an increased load on the CPU (sy - system time). At the same time, the vmstat executing in parallel will indicate that between 51% and 55% of the CPU time is being used for the execution of kernel code (for the generation of random numbers, in this case):

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  0 870968 164752   2556 440300   10   17   111   117  289  269 28 10 61  2  0
 3  0 870964 158576   2556 445440    0    0     0     0 2341 8240 19 53 28  0  0
 6  0 870960 154900   2556 450528    0    0    28     0 2022 6690 19 51 30  0  0
 3  1 870960 147560   2676 455800    0    0   232     8 2071 9254 20 55 25  0  0
 3  1 870956 142108   2824 462736    0    0   920  8580 1747 5959 15 55 28  2  0

As a high IO read load example, a large file (such as an ISO file) will be read and written to /dev/null using dd:

$ dd if=bigfile.iso of=/dev/null bs=1M

While reading, open another terminal and can execute vmstat in parallel. It will show the increased IO read load (the bi value - blocks received from a block device (blocks/s)):

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  2 882580 146512  44096 540636   14   17   123   115   48  461 28 10 60  1  0
 8  1 886024 142780  43864 547788    0 3448 48508  3448 3509 6549 18 18 25 38  0
 4  1 886028 101204  43864 590696   32   20 77344    20 2110 4447 10 19 40 32  0
 2  1 888788 110368  43784 585296    0 2760 52612  2968 3386 7110 17 18 24 41  0
 2  2 891308 110732  43784 591972   64 2528 85440  2528 2986 7273 18 24 32 25  0

To check linux distribution:

$ cat /etc/issue
Ubuntu 14.04.3 LTS \n \l

Or:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"

$ cat /etc/*-release
CentOS Linux release 7.2.1511 (Core) 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core) 
CentOS Linux release 7.2.1511 (Core) 

Or:

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

We can add a user to a wheel group:

$ sudo usermod -g wheel my-user-name

"The term wheel refers to a user account with a wheel bit, a system setting that provides additional special system privileges that empower a user to execute restricted commands that ordinary user accounts cannot access. The term is derived from the slang phrase big wheel, referring to a person with great power or influence." - Wheel (Unix term)

We may need to uncomment a line using sudo visudo and it should look like this:

## Allows people in group wheel to run all commands
%wheel  ALL=(ALL)       ALL

## Same thing without a password
%wheel  ALL=(ALL)       NOPASSWD: ALL

To create a user with home directory, we can use useradd:

$ sudo useradd -m testuser

To create a group:

$ sudo groupadd testgroup

We can add a user (for example, testuser) to a group (testgroup):

$ sudo usermod -aG testgroup testuser
$ groups testuser
testuser : testuser testgroup

$ sudo usermod -aG sudo testuser
$ groups testuser
testuser : testuser sudo testgroup

To remove a user from a group:

$ sudo gpasswd -d testuser sudo
Removing user testuser from group sudo
$ groups testuser
testuser : testuser testgroup

To delete a group:

$ sudo delgroup testgroup
Removing group `testgroup' ...

To delete a user:

$ sudo deluser testuser

Also check Managing User Account

We may want to check whether its inode change time (ctime) has changed from the last check:

$ stat testfile
  File: 'testfile'
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 802h/2050d	Inode: 27402256    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/       k)   Gid: ( 1000/       k)
Access: 2017-01-25 23:06:10.881876244 -0700
Modify: 2017-01-25 23:06:10.881876244 -0700
Change: 2017-01-25 23:06:10.881876244 -0700
 Birth: -

$ stat -c %Y a.txt
1487890881

$ touch a.txt

$ stat -c %Y a.txt
1487890975

$ stat -c %Y a.txt
1487890975

We changed the modification time via touch command.

Quite often, we need to kill a process when it behaves unfriendly. To list signals:

$ kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP
21) SIGTTIN	22) SIGTTOU	23) SIGURG	24) SIGXCPU	25) SIGXFSZ
26) SIGVTALRM	27) SIGPROF	28) SIGWINCH	29) SIGIO	30) SIGPWR
31) SIGSYS	34) SIGRTMIN	35) SIGRTMIN+1	36) SIGRTMIN+2	37) SIGRTMIN+3
38) SIGRTMIN+4	39) SIGRTMIN+5	40) SIGRTMIN+6	41) SIGRTMIN+7	42) SIGRTMIN+8
43) SIGRTMIN+9	44) SIGRTMIN+10	45) SIGRTMIN+11	46) SIGRTMIN+12	47) SIGRTMIN+13
48) SIGRTMIN+14	49) SIGRTMIN+15	50) SIGRTMAX-14	51) SIGRTMAX-13	52) SIGRTMAX-12
53) SIGRTMAX-11	54) SIGRTMAX-10	55) SIGRTMAX-9	56) SIGRTMAX-8	57) SIGRTMAX-7
58) SIGRTMAX-6	59) SIGRTMAX-5	60) SIGRTMAX-4	61) SIGRTMAX-3	62) SIGRTMAX-2
63) SIGRTMAX-1	64) SIGRTMAX	

Note that 15: SIGTERM is the default and it terminates a process.

To skill processes related, for example, nginx:

$ ps aux|grep nginx
k          674  0.0  0.0  22576   960 pts/23   S+   15:59   0:00 grep --color=auto nginx
root     10457  0.0  0.0 125096    72 ?        Ss   10:00   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 10458  0.0  0.0 125472  1620 ?        S    10:00   0:02 nginx: worker process
www-data 10459  0.0  0.0 125472  1608 ?        S    10:00   0:00 nginx: worker process

We can use killall:

$ sudo killall nginx

To kill processes belong a specific user, we use pkill:

$ sudo pkill -u username

To get the man page for signals:

$ man 7 signal

When we install packages with apt-get install or upgrade them with apt-get upgrade, the following information is obtained from our local system's configuration but not from the Internet:

1. what packages are available?
2. what versions?
3. where they can be retrieved from?

So, we need to run apt-get update before installing new packages as this updates the local repository information!

The apt-get update doesn't actually install new versions of software. It simply does update the Package Index!.

The APT package index is essentially a database of available packages from the repositories defined in the /etc/apt/sources.list file and in the /etc/apt/sources.list.d directory. So, to update the local package index with the latest changes made in the repositories, we simply use the following:

$ sudo apt-get update

At the command, it downloads the package lists from the repositories and "updates" them to get information on the newest versions of packages and their dependencies. It will do this for all repositories and PPAs.

Over time, updated versions of packages currently installed on our computer may become available from the package repositories. To upgrade packages of our system, first we need to update our package index as outlined above, and then type:

$ sudo apt upgrade

The command used for applying patches is the patch command but in order to apply a patch the diff command also needs to be used to supply the differences that need to be patched.

We have two files:

1. myscript.py:

   print("Hello World!")
   

   
   
2. myscript2.py:

   print("Hello World 2!")
   

Make a patch file:

$ diff -u myscript.py myscript2.py > mychange.patch

$ patch < mychange.patch
patching file myscript.py

We can check the myscript.py has been patched:

print("Hello World 2 !")

By default, xargs reads items from standard input and executes a command once for each argument. For example, we can create multiple directories in the following way:

$ echo "d1 d2 d3" | xargs mkdir

$ ls -d */
d1/	d2/	d3/ 

When we want to check the disk space for each directory, we use:

$ pwd
/var/log

$ du -hd1
724K	./displaypolicy
7.7M	./powermanagement
  0B	./timezone
4.9M	./asl
 81M	./DiagnosticMessages
  0B	./ppp
5.0M	./spinnaker
  0B	./apache2
  0B	./com.apple.xpc.launchd
  0B	./emond
 40M	./Bluetooth
 20K	./cups
  0B	./CoreDuet
  0B	./uucp
289M	.

We can get the same result using xargs:

$ ls -d */ | xargs du -sh
 40M	Bluetooth/
  0B	CoreDuet/
 81M	DiagnosticMessages/
  0B	apache2/
4.9M	asl/
  0B	com.apple.xpc.launchd/
 20K	cups/
724K	displaypolicy/
  0B	emond/
7.7M	powermanagement/
  0B	ppp/
5.0M	spinnaker/
  0B	timezone/
  0B	uucp/    

Note that the flag "-s" is equivalent to "-d 0". So, the xargs du -sh gets each directory from the piped input and spits out "du" for each directory, which makes the command equivalent to du -hd1.

We use the xargs with find command frequently. For example, if we want to delete files more than 3 years:

$ find ./ -type f -mtime +1095 | xargs rm

Note that we can do the same with -exec rm {}:

$ find ./ -type f -mtime +1095 -exec rm {} \;

In the command, {} is a placeholder for the result found by find, and \; says that for each found result, the "rm" command is executed once with the found result.

If we want to delete all the files at once not one by one, then we tell "find" and "xargs" to use null (ASCII NUL character '\000') separated instead of newline ('\n'):

$ find ./ -type f -mtime +1095 -print0 | xargs -0 rm

We can use it to replace a string recursively (for all files including sub-directories):

$ find . -type f -name "*.txt" -print0 | xargs -0 sed -i '' 's/abcde/12345/g'

1. find . -type f -name '*.txt' finds, in the current directory (.) and below, all regular files (-type f) whose names end in .txt
2. The "print0" puts ASCII NUL instead of '\n', and this requires "-0" after "xargs".
3. passes the output of that command (a list of filenames) to the next command
4. xargs gathers up those filenames and hands them one by one to sed
5. sed -i '' -e 's/abcde/12345/g' means "edit the file in place, without a backup (here, zero length string, '' is provides), and make the following substitution (s/abcde/12345) multiple times per line (/g)" (see man sed)

From How can I do a recursive find and replace from the command line?

We want to delete all buckets in aws using xargs:

$ aws s3 ls | awk '{print $3}' | xargs -n 1 -I% aws s3 rb s3://% --force

Here -I% and s3://% is to remove a space so that it can construct a command like "s3://myBucket" but not "s3:// mybucket". The -n 1 indicates we want to issue multiple commands (such as aws s3 rb arg1, aws s3 rb arg2) but not one command with multiple args (such as ars s3 rb arg1 arg2 ...).

We can delete all Docker images using xargs:

$ docker images | awk '{print $3}'
IMAGE
a76f8268cbae
10637d5d0d15
bfe77844ba19
584c892ed7f3
eb688f0facb9
25a54363c231
3f6ad1a4da3d
fb072f10bf15
...

$ docker images | awk '{print $3}' | xargs docker rmi
Untagged: youradddir_api:latest
Deleted: sha256:a76f8268cbaee278792955b70a9a604d8382f75081171601f30c8c8111194621
Deleted: sha256:8dc356607dded00215fc17b80555109ced239e7195e48b29edf11c67a92eec94
Deleted: sha256:899d5ae054646c50f05b6599a1a7f3d07b72cbb0e97eccdb6d16677d7650d55a
...

Want to delete all containers?

$  docker ps -a | awk '{print $1}' | xargs docker rm
1359e5cb29c3
cc2c089a5034
2687672d50c7
...

DevOps

DevOps / Sys Admin Q & A