DevOps / Sys Admin Q & A #13 : Is the Website down?

One of the first questions we want to answer when we can't reach a web server is whether the problem is on our end or on the remote end.

In this article, we assume that our problem is not on the network.

We can route to the machine, but we can't access the web server on port 80.

The next test is to see whether the port is even open. There are number of different ways to do this.

For one, we could try telnet:

$ telnet 54.153.94.154 80
Trying 54.153.94.154...

telnet: Unable to connect to remote host: Connection timed out

If we see "Connection refused", then either the port is down (most likely Apache/Nginx isn't running on the remote host or isn't listening on that port) or the firewall is blocking our access.

If telnet can connect, then, well, we don't have a networking problem at all.

If the web service is up but just isn't working the way we expect it to, we need to investigate our Apache configuration on the web server.

We may want to use nmap (Network Mapper) to test ports because it can often detect firewalls.

The output from nmap is a list of scanned targets, with supplemental information on each depending on the options used.
Key among that information is the "interesting ports table". That table lists the port number and protocol, service name, and state. The state is either open, filtered, closed, or unfiltered:

1. Open means that an application on the target machine is listening for connections/packets on that port.
2. Filtered means that a firewall, filter, or other network obstacle is blocking the port so that Nmap cannot tell whether it is open or closed. In other words, a filtered Nmap cannot determine whether the port is open because packet filtering prevents its probes from reaching the port. The filtering could be from a dedicated firewall device, router rules, or host-based firewall software...
3. Closed ports have no application listening on them, though they could open up at any time.
4. Ports are classified as unfiltered when they are responsive to Nmap's probes, but Nmap cannot determine whether they are open or closed.
5. The nmap reports the state combinations open|filtered and closed|filtered when it cannot determine which of the two states describe a port.

To test our server, we would type the following:

$ nmap -p 80 54.153.94.154

Starting Nmap 6.40 ( http://nmap.org ) at 2015-11-30 16:03 PST
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.27 seconds

If we open the port 80 on our Security group, we get the following output from nmap:

$ nmap -p 80 54.153.94.154

Starting Nmap 6.40 ( http://nmap.org ) at 2015-11-30 16:09 PST
Nmap scan report for ec2-54-153-94-154.us-west-1.compute.amazonaws.com (54.153.94.154)
Host is up (0.032s latency).
PORT   STATE SERVICE
80/tcp open http

Nmap done: 1 IP address (1 host up) scanned in 1.52 seconds

If our web server is not running, we get:

$ nmap -p 80 54.153.94.154

Starting Nmap 6.40 ( http://nmap.org ) at 2015-11-30 16:12 PST
Nmap scan report for ec2-54-153-94-154.us-west-1.compute.amazonaws.com (54.153.94.154)
Host is up (0.033s latency).
PORT   STATE  SERVICE
80/tcp closed http

Nmap done: 1 IP address (1 host up) scanned in 1.51 seconds

nmap is smart enough that it can often tell the difference between a closed port that is truly closed and a closed port behind a firewall. Now normally when a port is actually down, nmap will report it as closed.

filtered: nmap cannot determine whether the port is open because packet filtering prevents its probes from reaching the port. The filtering could be from a dedicated firewall device, router rules, or host-based firewall software.

The 2nd case of following two examples shows the "filtered" situation, and nmap cannot determine whether the port is open.

k@laptop:~$ nmap -p 80 45.79.90.218

Starting Nmap 6.40 ( http://nmap.org ) at 2015-12-17 09:21 PST
Nmap scan report for li1189-218.members.linode.com (45.79.90.218)
Host is up (0.030s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds

This 2nd case is the case when we run the server for Django using:

(sfprojectenv)[sfvue@sf sfproject]$ ./manage.py runserver 0.0.0.0:8080

On local machine, we issue the nmap command, and get the following:

k@laptop:~$ nmap -p 8080 45.79.90.218

Starting Nmap 6.40 ( http://nmap.org ) at 2015-12-17 09:21 PST
Nmap scan report for li1189-218.members.linode.com (45.79.90.218)
Host is up (0.031s latency).
PORT     STATE    SERVICE
8080/tcp filtered http-proxy

Nmap done: 1 IP address (1 host up) scanned in 0.42 seconds

On the browser, we get "This webpage is not available"

Even when we do not run the server, we get the same output from the nmap.

So, it looks like firewall problem

Once I open the port with this command:

[sfvue@sf sfproject]$ sudo iptables -I INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
[sfvue@sf sfproject]$ sudo service iptables save

we can see the port 8080 has been opened:

k@laptop:~$ nmap -p 8080 45.79.90.218

Starting Nmap 6.40 ( http://nmap.org ) at 2015-12-17 10:13 PST
Nmap scan report for li1189-218.members.linode.com (45.79.90.218)
Host is up (0.031s latency).
PORT     STATE SERVICE
8080/tcp open  http-proxy

Nmap done: 1 IP address (1 host up) scanned in 0.24 seconds

If we open up /etc/sysconfig/iptables, we can see the status of firewall settings. Actually, we could have edited the file directly to open up our port 8080:

...
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
...

Regarding iptables setup, please consult: Securing Your Server.

We may want to install zenmap to use GUI feature of nmap:

$ sudp apt-get install zenmap

zenmap lets us create an interactive network map based on information gathered by nmap:

Let's log in to our web server and test whether port 80 is listening:

$ netstat -lnp | grep 80
tcp        0      0    0.0.0.0:80              0.0.0.0:*               LISTEN      -               

The first column tells us what protocol the port is using. The second and third columns are the receive and send queues (both set to 0 here).

The column we want to pay attention to is the fourth column, as it lists the local address on which the host is listening. Here the 0.0.0.0:80 tells us that the host is listening on all of its IPs for port 80 traffic.

Among command-line tools used to interface with web APIs curl has an advantage over raw telnet for web server troubleshooting in that it takes care of the HTTP protocol for us and makes things like testing authentication, posting data, using SSL, and other functions we take for granted in a GUI web browser much easier.

$ curl http://www.example.net
<!doctype html>
<html>
<head>
...

When we're troubleshooting web server issues, the HTTP status code the web server returns for a request is invaluable source of information.

1xx: Informational Codes

2xx: Successful Codes

3xx: Redirection Codes

4xx: Client Error Codes

5xx: Server Error Codes

DevOps

DevOps / Sys Admin Q & A