Where to look first

When something breaks, the first job isn’t to fix it. It’s to find out which layer is broken — and the symptom usually tells you, if you read it exactly.

A request passes through several layers: DNS, the network, the load balancer, the server, the app, and the app’s dependencies. Each layer fails in its own way. So the exact thing the client sees tells you where to start.

The four symptoms

This is what each looks like from curl, the command-line tool for making requests:

curl: (28) Failed to connect to 10.0.11.24 port 8080 after 75004 ms: Timeout was reached

curl: (7) Failed to connect to 10.0.11.24 port 8080 after 2 ms: Couldn't connect to server

Newer versions of curl say “Couldn’t connect to server” for a refusal; add -v to see the words Connection refused. Notice the times, too: a timeout takes seconds or minutes; a refusal comes back almost instantly.

There’s one more kind of timeout. If the connection opens but the app never answers, curl says:

curl: (28) Operation timed out after 5006 milliseconds with 0 bytes received

Here the network worked. The app, or something the app is waiting on, is stuck. That’s a job for the logs, not the security groups.

What the load balancer’s 5xx codes mean

When the load balancer answers with a 5xx, the number tells you what went wrong between it and your servers:

The places to look, and how to get there

Target health. The load balancer checks every server and says why it’s unhappy with each one.

AWSIn the AWS console: EC2→Load Balancing→Target Groups→your target group→Targets

Recent changes. Most incidents follow a change. Check the last deploys, then ask CloudTrail for every change in the last hour.

AWSIn the AWS console: CloudTrail→Event history

Filter with Read only set to false. Who changed what shows how.

Logs. In Logs Insights, the first query shows the shape of the problem: when it started, when it stopped, how bad it is.

filter level = "ERROR"
| stats count() as errors by bin(1m), error_type

Metrics. On the load balancer’s Monitoring tab: are 5xx rising? Is response time up? Is it all servers or one? Logs and metrics explains which statistic to read.

A server that won’t boot. If it never becomes healthy, you often can’t open a shell on it either, because whatever broke the app broke that too. AWS can fetch the server’s boot output for you, without any network access to it:

AWSIn the AWS console: EC2→Instances→the instance→Actions→Monitor and troubleshoot→Get system log

It shows what the boot script printed and where it stopped. Get instance screenshot, in the same menu, shows the server’s screen.

Network ACLs: the other firewall

Security groups are the firewall you’ll use most, but each subnet also has a network ACL. The default one allows everything, so most people never touch it. If someone has changed it, it can cause timeouts that the security groups can’t explain.

The checklist

Work down it in order. Most problems are found by step 4.

Pick the symptom, answer the follow-up question, and see where it leads. Try every path: they’re the most common ways a web service breaks.

What to notice: the first choice — timeout, refused, 5xx or 4xx — already rules out most of the system. You never start by reading code. You start by reading the symptom exactly.

Free — everything runs on your own computer.

You’ll make each symptom happen on purpose, on your own machine, so you know them on sight. This lab reproduces the symptoms locally with a tiny web server; the same messages appear, with the same meanings, when the server is in AWS. About 15 minutes.

You need python3 and curl, which come with macOS and Ubuntu, including Ubuntu on WSL. You’ll use two terminal windows.

1. Make an empty folder and start a web server

   In window 1:

   TerminalCopy

   mkdir quickcart-triage

   Part	What it means
   mkdir	make a directory
   quickcart-triage	its name

   TerminalCopy

   cd quickcart-triage

   Part	What it means
   cd	move into a directory
   quickcart-triage	the empty folder you just made, so the server has nothing to share

   TerminalCopy

   python3 -m http.server 8080 --bind 127.0.0.1

   Part	What it means
   python3 -m http.server	run Python’s built-in web server, which serves the files in this folder
   8080	listen on port 8080
   --bind 127.0.0.1	only accept connections from this computer

   You should see

   Serving HTTP on 127.0.0.1 port 8080 (http://127.0.0.1:8080/) ...

   Leave it running. It prints a line for each request it receives.

2. A healthy request

   In window 2:

   TerminalCopy

   curl -i http://127.0.0.1:8080/

   Part	What it means
   curl	make a web request
   -i	also show the response’s status line and headers
   http://127.0.0.1:8080/	this computer, port 8080, the top page

   The first lines

   HTTP/1.0 200 OK
   Server: SimpleHTTP/0.6 Python/3.12.0

   Your Python version may differ. Below that is a short HTML page. 200: everything worked.

3. A 4xx: the request was wrong

   TerminalCopy

   curl -i http://127.0.0.1:8080/checkout

   Part	What it means
   curl -i	make a request and show the status line
   http://127.0.0.1:8080/checkout	a path that doesn’t exist on this server

   The first line

   HTTP/1.0 404 File not found

   The server is fine. It answered quickly and said the request asked for something that isn’t there. Restarting it would change nothing. In window 1, the server’s own log says the same: code 404, message File not found.

4. A 5xx: the server side failed

   TerminalCopy

   curl -i -X POST http://127.0.0.1:8080/

   Part	What it means
   curl -i	make a request and show the status line
   -X POST	use the POST method, like submitting a form. This little server can’t handle it
   http://127.0.0.1:8080/	the top page

   The first line

   HTTP/1.0 501 Unsupported method ('POST')

   A 5xx: the request reached the server, and the server couldn’t do it. For a real app, this is when you read its logs, which is exactly what window 1 now shows: code 501, message Unsupported method ('POST').

5. Refused: nothing is listening

   In window 1, stop the server with Control+C. Then in window 2:

   TerminalCopy

   curl -v http://127.0.0.1:8080/

   Part	What it means
   curl	make a web request
   -v	verbose: show each step of the connection, including why it failed
   http://127.0.0.1:8080/	the same address as before

   Among the lines it prints

   * connect to 127.0.0.1 port 8080 from 127.0.0.1 port 59283 failed: Connection refused
   * Failed to connect to 127.0.0.1 port 8080 after 0 ms: Couldn't connect to server
   curl: (7) Failed to connect to 127.0.0.1 port 8080 after 0 ms: Couldn't connect to server

   Your port numbers will differ. 0 ms: your computer answered at once that nothing is on port 8080. The network is fine. The process is gone.

6. Timed out: nothing answers at all

   TerminalCopy

   curl --connect-timeout 5 http://10.255.255.1:8080/

   Part	What it means
   curl	make a web request
   --connect-timeout 5	give up if the connection hasn’t opened after 5 seconds, instead of waiting a minute or more
   http://10.255.255.1:8080/	a private address that nothing on a normal home or office network uses, so nothing will answer

   You should see

   curl: (28) Failed to connect to 10.255.255.1 port 8080 after 5006 ms: Timeout was reached

   5006 ms: curl waited the whole five seconds and heard nothing — not even a refusal. This is what a security group dropping traffic looks like from outside. On some networks you may see No route to host instead; that’s also the network.

7. Match each symptom to its layer

   Without looking back, fill in the right-hand column on paper, then check against the idea:

   What you saw	Layer to check first
   404 File not found, instantly
   501, instantly, and the server logged it
   Couldn't connect, Connection refused, 0 ms
   Timeout was reached after 5,006 ms

   You should see

   The request itself; the app and its logs; the process and its port; the network path: security groups, routes and network ACLs.

8. Clean-up step: Clean up

   The server is already stopped. Remove the empty folder from window 1:

   TerminalCopy

   cd ..

   Part	What it means
   cd ..	move up one folder, out of quickcart-triage

   TerminalCopy

   rmdir quickcart-triage

   Part	What it means
   rmdir	remove a directory, which only works if it’s empty
   quickcart-triage	the folder you made

   You should see

   No output. Nothing is left running, and nothing was created in AWS.