DevOps / Sys Admin Q & A: Linux Systemd - journalctl
bogotobogo.com site search:
Introduction
journalctl is a command to view logs collected by systemd.
It's a command dedicated to logging that systemd provides.
The systemd-journald service is responsible for systemd's log collection, and it retrieves messages from the kernel and systemd services.
Since journald stores log data in a binary format instead of a plaintext format,
journalctl is the standard way of reading log messages processed by journald.
journalctl command
Running the journalctl command without any arguments spits all the logs in our journal, which is most likely thousands of lines.
journalctl -u
To see the logs for a systemd service, we need to pass the name of a systemd unit with the -u option, in this case, sshd:
$ journalctl -u sshd -- Logs begin at Sun 2020-08-09 03:25:05 UTC, end at Sun 2020-08-09 03:26:10 UTC. -- Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon... Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1094]: Server listening on 0.0.0.0 port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1094]: Server listening on :: port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Stopping OpenSSH server daemon... Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Stopped OpenSSH server daemon. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon... Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1105]: Server listening on 0.0.0.0 port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1105]: Server listening on :: port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon. Aug 09 03:26:10 ip-172-31-5-192.ec2.internal sshd[1126]: Accepted publickey for centos from 73.70.219.237 port 61127 ssh2: RSA SHA256:NQp2twy8c9Lea
Compared with the following output from systemctl, we can see journalctl gives the more info:
$ systemctl status sshd
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-08-09 03:26:04 UTC; 1min 20s ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 1105 (sshd)
CGroup: /system.slice/sshd.service
└─1105 /usr/sbin/sshd -D
Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon...
Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1105]: Server listening on 0.0.0.0 port 22.
Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1105]: Server listening on :: port 22.
Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon.
Aug 09 03:26:10 ip-172-31-5-192.ec2.internal sshd[1126]: Accepted publickey for centos from 73.70.219.237 port 61127 ssh2: RSA SHA256:NQp2...QfU3j0
Hint: Some lines were ellipsized, use -l to show in full.
We can use it similar to the tail -f:
$ journalctl -fu sshd -- Logs begin at Sun 2020-08-09 03:25:05 UTC. -- Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1094]: Server listening on :: port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Stopping OpenSSH server daemon... Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Stopped OpenSSH server daemon. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon... Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1105]: Server listening on 0.0.0.0 port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal sshd[1105]: Server listening on :: port 22. Aug 09 03:26:04 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon. Aug 09 03:26:10 ip-172-31-5-192.ec2.internal sshd[1126]: Accepted publickey for centos from 73.70.219.237 port 61127 ssh2: RSA SHA256:NQp2twy8c9Leaht4Z0r7Whgpr97wIhLJB6kFIQfU3j0 Aug 09 03:30:38 ip-172-31-5-192.ec2.internal sshd[1160]: Accepted publickey for centos from 73.70.219.237 port 61143 ssh2: RSA SHA256:NQp2twy8c9Leaht4Z0r7Whgpr97wIhLJB6kFIQfU3j0 Aug 09 03:32:39 ip-172-31-5-192.ec2.internal systemd[1]: Stopping OpenSSH server daemon... Aug 09 03:32:39 ip-172-31-5-192.ec2.internal systemd[1]: Stopped OpenSSH server daemon. Aug 09 03:32:39 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon... Aug 09 03:32:39 ip-172-31-5-192.ec2.internal sshd[1220]: Server listening on 0.0.0.0 port 22. Aug 09 03:32:39 ip-172-31-5-192.ec2.internal sshd[1220]: Server listening on :: port 22. Aug 09 03:32:39 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon. Aug 09 03:32:51 ip-172-31-5-192.ec2.internal systemd[1]: Stopping OpenSSH server daemon... Aug 09 03:32:51 ip-172-31-5-192.ec2.internal systemd[1]: Stopped OpenSSH server daemon. Aug 09 03:32:51 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon... Aug 09 03:32:51 ip-172-31-5-192.ec2.internal sshd[1229]: Server listening on 0.0.0.0 port 22. Aug 09 03:32:51 ip-172-31-5-192.ec2.internal sshd[1229]: Server listening on :: port 22. Aug 09 03:32:51 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon. Aug 09 03:32:52 ip-172-31-5-192.ec2.internal systemd[1]: Stopping OpenSSH server daemon... Aug 09 03:32:52 ip-172-31-5-192.ec2.internal systemd[1]: Stopped OpenSSH server daemon. Aug 09 03:32:52 ip-172-31-5-192.ec2.internal systemd[1]: Starting OpenSSH server daemon... Aug 09 03:32:52 ip-172-31-5-192.ec2.internal sshd[1237]: Server listening on 0.0.0.0 port 22. Aug 09 03:32:52 ip-172-31-5-192.ec2.internal sshd[1237]: Server listening on :: port 22. Aug 09 03:32:52 ip-172-31-5-192.ec2.internal systemd[1]: Started OpenSSH server daemon.
Got the above output by restarting sshd couple of times.
Interestingly, if done it too many times, we got the following in the journal log:
... Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Unit sshd.service entered failed state. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed.
The console output from systemctl looks like this:
# systemctl restart sshd Job for sshd.service failed because start of the service was attempted too often. See "systemctl status sshd.service" and "journalctl -xe" for details. To force a start use "systemctl reset-failed sshd.service" followed by "systemctl start sshd.service" again.
We can limit the lines of output using -n:
# journalctl -n 5 -u sshd -- Logs begin at Sun 2020-08-09 03:25:01 UTC, end at Sun 2020-08-09 03:56:01 UTC. -- Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed.
journalctl -r
Output in reverse order (latest first):
# journalctl -r -u sshd -- Logs begin at Sun 2020-08-09 03:25:01 UTC, end at Sun 2020-08-09 04:01:01 UTC. -- Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Unit sshd.service entered failed state. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service
journalctl -p
We can display the output what we want, here, preferences, such as "err", "warning", or "info":
# journalctl -p err -u sshd -- Logs begin at Sun 2020-08-09 03:25:01 UTC, end at Sun 2020-08-09 04:01:01 UTC. -- Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. # journalctl -p warning -u sshd -- Logs begin at Sun 2020-08-09 03:25:01 UTC, end at Sun 2020-08-09 04:01:01 UTC. -- Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: start request repeated too quickly for sshd.service Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: Failed to start OpenSSH server daemon. Aug 09 03:35:07 ip-172-31-5-192.ec2.internal systemd[1]: sshd.service failed.
DevOps
DevOps / Sys Admin Q & A
Linux - system, cmds & shell
- Linux Tips - links, vmstats, rsync
- Linux Tips 2 - ctrl a, curl r, tail -f, umask
- Linux - bash I
- Linux - bash II
- Linux - Uncompressing 7z file
- Linux - sed I (substitution: sed 's///', sed -i)
- Linux - sed II (file spacing, numbering, text conversion and substitution)
- Linux - sed III (selective printing of certain lines, selective definition of certain lines)
- Linux - 7 File types : Regular, Directory, Block file, Character device file, Pipe file, Symbolic link file, and Socket file
- Linux shell programming - introduction
- Linux shell programming - variables and functions (readonly, unset, and functions)
- Linux shell programming - special shell variables
- Linux shell programming : arrays - three different ways of declaring arrays & looping with $*/$@
- Linux shell programming : operations on array
- Linux shell programming : variables & commands substitution
- Linux shell programming : metacharacters & quotes
- Linux shell programming : input/output redirection & here document
- Linux shell programming : loop control - for, while, break, and break n
- Linux shell programming : string
- Linux shell programming : for-loop
- Linux shell programming : if/elif/else/fi
- Linux shell programming : Test
- Managing User Account - useradd, usermod, and userdel
- Linux Secure Shell (SSH) I : key generation, private key and public key
- Linux Secure Shell (SSH) II : ssh-agent & scp
- Linux Secure Shell (SSH) III : SSH Tunnel as Proxy - Dynamic Port Forwarding (SOCKS Proxy)
- Linux Secure Shell (SSH) IV : Local port forwarding (outgoing ssh tunnel)
- Linux Secure Shell (SSH) V : Reverse SSH Tunnel (remote port forwarding / incoming ssh tunnel) /)
- Linux Processes and Signals
- Linux Drivers 1
- tcpdump
- Linux Debugging using gdb
- Embedded Systems Programming I - Introduction
- Embedded Systems Programming II - gcc ARM Toolchain and Simple Code on Ubuntu/Fedora
- LXC (Linux Container) Install and Run
- Linux IPTables
- Hadoop - 1. Setting up on Ubuntu for Single-Node Cluster
- Hadoop - 2. Runing on Ubuntu for Single-Node Cluster
- ownCloud 7 install
- Ubuntu 14.04 guest on Mac OSX host using VirtualBox I
- Ubuntu 14.04 guest on Mac OSX host using VirtualBox II
- Windows 8 guest on Mac OSX host using VirtualBox I
- Ubuntu Package Management System (apt-get vs dpkg)
- RPM Packaging
- How to Make a Self-Signed SSL Certificate
- Linux Q & A
- DevOps / Sys Admin questions
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization
My YouTube channel