DevOps / Sys Admin Q & A #22 : lsof

bogotobogo.com site search:

Introduction

lsof stands for "list open files", which is used in many Unix-like systems to report a list of all open files and the processes that opened them.

# lsof
COMMAND     PID   TID       USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
...
upstart-d  2902                k    6u     unix 0x0000000000000000       0t0      29363 type=STREAM
fcitx      2908                k  cwd       DIR                8,2      4096          2 /
fcitx      2908                k  rtd       DIR                8,2      4096          2 /
upstart-f  2915                k    4r     FIFO               0,10       0t0      29346 pipe
...

Let's look at the header line. 99% of time, we're interested in the COMMAND, PID, and the lst column, NAME which shows the path. Though the header labels are self explanatory, the followings are the descriptions of the fields:

FD - Represents the file descriptor. Some of the values of FDs are:

cwd - Current Working Directory
txt - Text file
mem - Memory mapped file
mmap - Memory mapped device
NUMBER - Represent the actual file descriptor. The character after the number i.e '1u', represents the mode in which the file is opened. r for read, w for write, u for read and write.

TYPE - Specifies the type of the file. Some of the values of TYPEs are:

REG - Regular File
DIR - Directory
FIFO - First In First Out
CHR - Character special file

List processes which opened a specific file

We can list only the processes which opened a specific file, by providing the filename as arguments. For example, to see which process opened "/var/log/httpd/access.log" file:

# lsof /var/log/httpd/access.log
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
httpd    8379 apache   43w   REG  202,0    96050 237240 /var/log/httpd/access.log
httpd   14976 apache   43w   REG  202,0    96050 237240 /var/log/httpd/access.log
httpd   17680 apache   43w   REG  202,0    96050 237240 /var/log/httpd/access.log
...

As we can see from the output, Apache is using the file: the command is "httpd" and the user is "apache". Note that we have "w" in FD column, and we know the "httpd" only care about writing.

List opened files for a specific process

Let's check what files are opened by the 2nd process in the previous section:

# lsof -p 14976
COMMAND   PID   USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
httpd   14976 apache  cwd       DIR              202,0     4096         2 /
httpd   14976 apache  rtd       DIR              202,0     4096         2 /
httpd   14976 apache  txt       REG              202,0   507000      9287 /usr/sbin/httpd
httpd   14976 apache  mem       REG              202,0    57952      4690 /usr/lib64/libzip.so.2.1.0
httpd   14976 apache  mem       REG              202,0    58376    247884 /usr/lib64/php/modules/zip.so
...
httpd   14976 apache    1u     unix 0xffff8800023ef400      0t0 161426920 socket
httpd   14976 apache    2w      REG              202,0   179935    237243 /var/log/httpd/error_log
httpd   14976 apache    3u     sock                0,7      0t0 161426928 protocol: TCP
httpd   14976 apache    4u     IPv6          161426929      0t0       TCP *:http (LISTEN)
httpd   14976 apache    5u     sock                0,7      0t0 161426936 protocol: TCP
httpd   14976 apache    6u     IPv6          161426937      0t0       TCP *:https (LISTEN)
...

The file type 'txt' is the actual program, the "so' files are shared libraries, and we have a writing to "error_log" with "2w" where the "2" means "stderr". Also, we have "stdout" to "socket" with FD "1u".

To look for the binary:

# lsof -p 14976 | grep bin
httpd   14976 apache  txt       REG              202,0   507000      9287 /usr/sbin/httpd

Which file is the process is logging to?

# lsof -p 14976 | grep log
httpd   14976 apache  mem       REG              202,0    11208    293352 /usr/lib64/httpd/modules/mod_logio.so
...
httpd   14976 apache   24w      REG              202,0    56699    237244 /var/log/httpd/error.log
...
httpd   14976 apache   43w      REG              202,0    98016    237240 /var/log/httpd/access.log
httpd   14976 apache   44w      REG              202,0     3732    237245 /var/log/httpd/ssl_access_log
httpd   14976 apache   45w      REG              202,0     4827    237247 /var/log/httpd/ssl_request_log

List opened files under a directory

We can list the processes which opened files under a specified directory using +D option. +D will recurse the sub directories also. If we don't want lsof to recurse, then use +d option.

# lsof /var/log
COMMAND    PID     USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
rsyslogd   808   syslog    5w   REG    8,2     2269 8782114 /var/log/syslog
rsyslogd   808   syslog    6w   REG    8,2    62336 8784023 /var/log/auth.log
mysqld    1027    mysql    1w   REG    8,2        0 8781897 /var/log/mysql/error.log
mysqld    1027    mysql    2w   REG    8,2        0 8781897 /var/log/mysql/error.log
lightdm   1054     root    6w   REG    8,2        0 8783083 /var/log/lightdm/lightdm.log
Xorg      1096     root    0w   REG    8,2    60398 8781901 /var/log/Xorg.0.log
Xorg      1096     root    2w   REG    8,2        0 8783085 /var/log/lightdm/x-0.log
...

List opened files based on process names starting with

We can list the files opened by process names starting with a string, using -c option. -c followed by the process name will list the files opened by the process starting with that processes name. We can give multiple -c switch on a single command line.

# lsof -c ssh -c init
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
sshd    992 root  cwd    DIR                8,2     4096        2 /
sshd    992 root  rtd    DIR                8,2     4096        2 /
sshd    992 root  txt    REG                8,2   799216 25701529 /usr/sbin/sshd
sshd    992 root  mem    REG                8,2    47600  6949987 /lib/x86_64-linux-gnu/libnss_files-2.23.so
sshd    992 root  mem    REG                8,2    47648  6949984 /lib/x86_64-linux-gnu/libnss_nis-2.23.so
...

List all network connections

If no address is specified, this option selects the listing of all Internet and x.25 (HP-UX) network files:

# lsof -i
COMMAND     PID     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
cups-brow   793     root    8u  IPv4   20965      0t0  UDP *:ipp 
astxdaemo   820     root    9u  IPv4   22498      0t0  TCP *:55920 (LISTEN)
mysqld     1027    mysql   18u  IPv4   20440      0t0  TCP localhost:mysql (LISTEN)
postgres   1099 postgres    6u  IPv4   21038      0t0  TCP localhost:postgresql (LISTEN)
postgres   1099 postgres   10u  IPv4   20223      0t0  UDP localhost:38283->localhost:38283 
nginx      1124     root    6u  IPv4   20979      0t0  TCP *:http (LISTEN)
...

We get a similar output from netstat -an:

# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN         
tcp        0      0 0.0.0.0:55920           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:42233           0.0.0.0:*               LISTEN     
tcp        0      0 192.168.1.1:44902       151.101.40.64:443       ESTABLISHED

Which process is listening on a specific port?

For port 80:

# lsof -i :80
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
httpd    8379 apache    4u  IPv6 161426929      0t0  TCP *:http (LISTEN)
httpd   14976 apache    4u  IPv6 161426929      0t0  TCP *:http (LISTEN)
...

Only want to see ports in the LISTEN state?

$ sudo lsof -PiTCP -sTCP:LISTEN
com.docke   646 kihyuckhong   18u  IPv4 0x38d0f1345a33abad      0t0  TCP *:8081 (LISTEN)
com.docke   646 kihyuckhong   19u  IPv4 0x38d0f13461d97bad      0t0  TCP *:8087 (LISTEN)
com.docke   646 kihyuckhong   20u  IPv6 0x38d0f1344efcb605      0t0  TCP localhost:8087 (LISTEN)
com.docke   646 kihyuckhong   21u  IPv6 0x38d0f1345d161185      0t0  TCP localhost:8081 (LISTEN)
com.docke   646 kihyuckhong   22u  IPv4 0x38d0f1345d02312d      0t0  TCP *:8080 (LISTEN)
com.docke   646 kihyuckhong   23u  IPv6 0x38d0f1345d15e385      0t0  TCP localhost:8080 (LISTEN)
cupsd      1133        root    5u  IPv6 0x38d0f1344efca4c5      0t0  TCP localhost:631 (LISTEN)
cupsd      1133        root    6u  IPv4 0x38d0f1344288b7cd      0t0  TCP localhost:631 (LISTEN)
VBoxHeadl  3783 kihyuckhong   16u  IPv4 0x38d0f1345c88fbad      0t0  TCP localhost:59562 (LISTEN)
...

Note that without the sudo, lsof can only see processes we own, and hence won't show any ports opened by system.

processes.

Which process is using a specific protocol?

UDP:

# lsof -i udp
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
chronyd    2601   chrony    1u  IPv4  11038      0t0  UDP localhost:323 
chronyd    2601   chrony    2u  IPv6  11039      0t0  UDP localhost:323 
avahi-dae  2606    avahi   12u  IPv4  11459      0t0  UDP *:mdns 
avahi-dae  2606    avahi   13u  IPv4  11460      0t0  UDP *:33059 
postgres   3027 postgres   10u  IPv6  14718      0t0  UDP localhost:48365->localhost:48365
...

List files opened by a specific user

To find the list of files opened by a specific users, use -u option: