DevOps / Sys Admin Q & A : Troubleshooting 5xx server errors
bogotobogo.com site search:
502 Bad Gateway
This error is usually due to improperly configured proxy (edge) servers. So, it cannot get a valid (or any) response from the origin (upstream) server:
Picture credit How to Solve 502 Bad Gateway Issues?
In general there are couple of reasons for the 502 Bad Gateway error:
- Origin server down: The server is not reachable either because it is down or there is no connectivity to the server.
- DNS: The domain name is not resolving to the correct IP or it does not resolve to any IP.
- Firewall blocks request: A firewall blocks the communication between the edge servers and the origin server. We may want to check firewall logs (sudo tail -f /var/log/messages) if there are any dropped requests.
- Server overload - a server can crash if it has exhausted its memory, due to high traffic.
- If the server in question is a reverse proxy server (a load balancer), we can try the followings:
- The backend servers (where the HTTP requests are being forwarded to) are healthy.
- The reverse proxy is configured properly, with the proper backends specified.
- The network connection between the backend servers and reverse proxy server is healthy. If the servers can communicate on other ports, make sure that the firewall is allowing the traffic between them.
- If our web application is configured to listen on a port, ensure that our firewall is blocking it.
- The client's cache - Though this is a server side error, but hard-refreshing a browser (Cmd + Shift + R on Macs) may result in a different proxy being used, and resolve the web server's content. This could be extremely temporary, so trying the page reload will often be successful. If this works, the problem was due to a temporarily overloaded server.
503 Service Unavailable
The 503 status code indicates that the site is still functional, but it's temporarily unavailable. The typical causes:
- Server in maintenance - Visitors will see a 503 HTTP error message if the site is offline for routine updates.
- Server Overloaded - If the site is receiving more traffic than the server can handle while malware, attacks or incorrect programming can cause 503 HTTP errors as well.
- Client Side Issues - Occasionally, visitors may get this error if their client-side DNS server goes down or is improperly configured.
504 Gateway Timeout
A 504 Gateway Timeout indicates that the proxy/edge server did not get a response from the upstream/origin server in a given time window.
Picture credit HTTP Status Code 504 Gateway Timeout
Fix:
- DNS FQDN check - either by online domain name check (i.e. site.com) or using a command such as "nslookup site.com".
- The origin server is down or not working properly - Check connection using traceroute/ping command either on local or origin-server.
- There might be some miss communication or drop connection between back-end servers and a load balancer. Tweek load balancer idle timeout or server configurations such as KeepAliveTimeout or Timeout, or enable KeepAlive. Usually, the Timeouts of the server should be longer than the load balancer's idle timeout.
- One more, related to the load balancer, especially, "Connection Draining". It should be enabled and set to at least 60 seconds. When auto-scale instances are terminating, we will get site cannot be reached error if the draining value exceeding the "KeepAliveTimeout".
DevOps
DevOps / Sys Admin Q & A
Linux - system, cmds & shell
- Linux Tips - links, vmstats, rsync
- Linux Tips 2 - ctrl a, curl r, tail -f, umask
- Linux - bash I
- Linux - bash II
- Linux - Uncompressing 7z file
- Linux - sed I (substitution: sed 's///', sed -i)
- Linux - sed II (file spacing, numbering, text conversion and substitution)
- Linux - sed III (selective printing of certain lines, selective definition of certain lines)
- Linux - 7 File types : Regular, Directory, Block file, Character device file, Pipe file, Symbolic link file, and Socket file
- Linux shell programming - introduction
- Linux shell programming - variables and functions (readonly, unset, and functions)
- Linux shell programming - special shell variables
- Linux shell programming : arrays - three different ways of declaring arrays & looping with $*/$@
- Linux shell programming : operations on array
- Linux shell programming : variables & commands substitution
- Linux shell programming : metacharacters & quotes
- Linux shell programming : input/output redirection & here document
- Linux shell programming : loop control - for, while, break, and break n
- Linux shell programming : string
- Linux shell programming : for-loop
- Linux shell programming : if/elif/else/fi
- Linux shell programming : Test
- Managing User Account - useradd, usermod, and userdel
- Linux Secure Shell (SSH) I : key generation, private key and public key
- Linux Secure Shell (SSH) II : ssh-agent & scp
- Linux Secure Shell (SSH) III : SSH Tunnel as Proxy - Dynamic Port Forwarding (SOCKS Proxy)
- Linux Secure Shell (SSH) IV : Local port forwarding (outgoing ssh tunnel)
- Linux Secure Shell (SSH) V : Reverse SSH Tunnel (remote port forwarding / incoming ssh tunnel) /)
- Linux Processes and Signals
- Linux Drivers 1
- tcpdump
- Linux Debugging using gdb
- Embedded Systems Programming I - Introduction
- Embedded Systems Programming II - gcc ARM Toolchain and Simple Code on Ubuntu/Fedora
- LXC (Linux Container) Install and Run
- Linux IPTables
- Hadoop - 1. Setting up on Ubuntu for Single-Node Cluster
- Hadoop - 2. Runing on Ubuntu for Single-Node Cluster
- ownCloud 7 install
- Ubuntu 14.04 guest on Mac OSX host using VirtualBox I
- Ubuntu 14.04 guest on Mac OSX host using VirtualBox II
- Windows 8 guest on Mac OSX host using VirtualBox I
- Ubuntu Package Management System (apt-get vs dpkg)
- RPM Packaging
- How to Make a Self-Signed SSL Certificate
- Linux Q & A
- DevOps / Sys Admin questions
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization