Security groups
The firewall on every resource, and chaining them.
Part 1: The problem
Priya tidied up QuickCart’s security groups one evening. On quickcart-app-sg she removed the outbound rule. The app only ever answered requests, she reasoned, so why would it need to send anything out?
Nothing broke straight away. The running servers kept serving orders. But the next morning the auto scaling group started new servers, and none of them came up.
The auto scaling group launched a server, waited, marked it unhealthy, terminated it, and launched another. All morning. Session Manager couldn’t connect to any of them. The load balancer said they failed their health checks. Nobody could see why.
Leo finally opened the console log of one doomed server with Actions → Monitor and troubleshoot → Get system log. The start-up script was stuck on dnf update. It wasn’t failing, just waiting. So was the Session Manager agent. Neither could reach the internet, because a server with no outbound rule can’t start any connection at all.
The app didn’t need to send anything to customers. But a server does plenty of things on its own: it downloads updates, installs the app, and reports in to AWS. Every one of those is a connection the server starts.
This lesson is about getting those rules right: what a security group allows, what it quietly drops, and how to tell a network problem from a code problem in seconds.
Part 2: The idea
A security group is a firewall attached to one resource: a server, a load balancer, a database. It’s a list of rules saying what may connect to that resource, and what the resource may connect to.
The route table decides whether traffic can find a path to a subnet. The security group decides whether the resource lets it in. Both have to say yes.
Four rules about how security groups behave
| Behaviour | What it means | Why it matters |
|---|---|---|
| Allow only | Every rule allows something. There are no deny rules | Anything not allowed is blocked. You never write “block this” |
| Stateful | Once a connection is allowed in, its replies are allowed out, automatically. The same is true the other way round | You only write rules for connections that start. Replies take care of themselves |
| Silent | Blocked traffic is dropped without a reply. The sender hears nothing | The sender waits until it gives up. This is why blocked traffic shows up as a time-out |
| Immediate | A rule change applies within seconds, to every resource using the group | No restart. Also no undo button, so change rules carefully |
What’s in a rule
Each rule has three parts:
| Part | Example | What it means |
|---|---|---|
| Protocol | TCP | The kind of traffic. Nearly everything you’ll write is TCP |
| Port | 8080 | Which door on the resource. A web server might listen on 443, an app on 8080, PostgreSQL on 5432 |
| Source (inbound) or destination (outbound) | quickcart-alb-sg | Who may connect. An address range such as 0.0.0.0/0, or another security group |
What a new security group allows
| Direction | A new security group starts with | So |
|---|---|---|
| Inbound | No rules | Nothing can connect in until you add a rule |
| Outbound | One rule: all traffic to 0.0.0.0/0 | The resource can start connections to anywhere |
That outbound rule is the one Priya deleted. It looks unnecessary, which is why people remove it.
Name the group before, not an address range
A source can be another security group. The rule then means anything that has that security group attached, wherever it is and whatever its address.
That’s how QuickCart chains its three tiers, each one letting in only the one in front of it:
| Security group | Attached to | Inbound rule | Why |
|---|---|---|---|
quickcart-alb-sg | The load balancer | HTTPS, port 443, from 0.0.0.0/0 | Customers can be anywhere |
quickcart-app-sg | The app servers | TCP, port 8080, from quickcart-alb-sg | Only the load balancer may reach the app |
quickcart-db-sg | The database | PostgreSQL, port 5432, from quickcart-app-sg | Only the app may reach the database |
If the auto scaling group adds ten servers, the database rule needs no change: the new servers carry quickcart-app-sg, so they’re allowed. And if someone puts a test server in the app subnet without that group, it can’t reach the database, even though its address is in the same range.
Time-out or error: the fastest diagnosis you’ll ever make
| What you see | What happened | Where to look |
|---|---|---|
| Waits, then times out | Packets were silently dropped. Nothing answered | Security groups, route tables, network ACLs |
| Connection refused, straight away | The packet arrived, but nothing is listening on that port | Is the app running? Is it listening on that port? |
| An HTTP error such as 500, straight away | The app received the request and failed | The app’s logs |
Behind a load balancer the same rule holds. The load balancer tries to connect to a server for 10 seconds, then returns 504 Gateway Timeout. A 504 with nothing in the app’s logs almost always means a security group.
Network ACLs, briefly
A VPC has a second, older firewall: the network ACL. You’ll rarely change it, but you should recognise it.
| Security group | Network ACL | |
|---|---|---|
| Attached to | One resource | A whole subnet |
| Rules | Allow only | Allow and deny |
| Order | No order. All rules are checked | Numbered. The lowest-numbered match wins |
| Replies | Stateful: replies are automatic | Stateless: replies need their own rule |
| Default | New group: nothing in, everything out | The VPC’s default ACL allows everything both ways |
Because a network ACL is stateless, a reply needs its own rule. Replies go back to the random high port the other side picked, called an ephemeral port, usually somewhere between 1024 and 65535. Forgetting that return rule is the classic network ACL mistake. QuickCart leaves the default network ACL alone and does all its filtering with security groups.
Part 3: See it
Below is QuickCart’s chain: load balancer, app, database, each with its own security group. Send a request with every rule right. Then break one rule at a time, send again, and watch the clock at the bottom. Finally, try a bug in the code instead.
What to notice: every broken rule gave the same result: a long wait, then a time-out, and nothing in the app’s logs about why. The bug in the code failed in a fraction of a second. How long a failure takes tells you where to look.
Part 4: Do it
Pennies — security groups are free. The test server costs about two cents an hour while it runs, including its public IP address (prices change). You delete it at the end.
You’ll build QuickCart’s three-group chain, then launch one small server and see a time-out, a success and a refused connection with your own eyes. About 30 minutes.
Check the region selector at the top right says N. Virginia (us-east-1). The lab uses your account’s default VPC, which AWS creates in every region. If you deleted it, create the VPC from the VPC lesson and choose it instead, with a public subnet.
Create the load balancer's security group
In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group
Field Enter Security group name quickcart-alb-sgDescription QuickCart load balancer: HTTPS from anywhereVPC your default VPC Under Inbound rules, choose Add rule:
Type Source HTTPS (port 443 fills in) Anywhere-IPv4 ( 0.0.0.0/0)Leave Outbound rules as they are: one rule, All traffic to
0.0.0.0/0. Choose Create security group.You should seeThe group’s details page, with one inbound rule and one outbound rule.
Create the app's security group, pointing at the first
In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group
Field Enter Security group name quickcart-app-sgDescription QuickCart app servers: 8080 from the load balancer onlyVPC your default VPC Under Inbound rules, choose Add rule:
Type Port range Source Custom TCP 8080Custom, then start typing quickcart-alb-sgand pick it from the listLeave the outbound rule. Choose Create security group.
You should seeThe inbound rule’s source shows the ID of
quickcart-alb-sg, startingsg-, instead of an address range.Create the database's security group, pointing at the app
In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group
Field Enter Security group name quickcart-db-sgDescription QuickCart database: PostgreSQL from the app onlyVPC your default VPC Type Source PostgreSQL (port 5432 fills in) Custom, then pick quickcart-app-sgChoose Create security group. That’s the whole chain: three groups, three inbound rules, and no address ranges except the front door.
Create a test group with no inbound rules
Now you’ll see the three outcomes for yourself. Create one more group:
In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group
Field Enter Security group name quickcart-lab-sgDescription Lab: test time-outs from my laptopVPC your default VPC Add no inbound rules. Choose Create security group.
Launch a small server running a web server on port 8080
In the AWS console: EC2InstancesLaunch instances
Field Enter Name quickcart-labApplication and OS Images Amazon Linux 2023 AMI (the default) Instance type t3.micro Key pair Proceed without a key pair Network settings → Edit → Auto-assign public IP Enable Firewall (security groups) Select existing security group, then quickcart-lab-sgOpen Advanced details. In the User data box at the bottom, paste this script. It runs once, as the server first starts:
user data#!/bin/bash systemd-run --unit quickcart-web /usr/bin/python3 -m http.server 8080Part What it means #!/bin/bashrun this script with bash systemd-runstart a program as a background service, so it keeps running after the script ends --unit quickcart-webthe service’s name /usr/bin/python3 -m http.serverPython’s built-in web server. Amazon Linux 2023 includes Python 8080the port it listens on, like QuickCart’s app Choose Launch instance. Wait until the instance shows Running and its status check shows 2/2 checks passed, about two minutes. Then select it and copy its Public IPv4 address.
Try it with no rule: a time-out
On your computer, open Terminal and run the command below. Replace the highlighted address with the public IPv4 address you copied:
Terminalcurl --max-time 5 http://203.0.113.10:8080Part What it means curlmake a web request from the terminal and print the answer --max-time 5give up after 5 seconds instead of waiting for minutes http://203.0.113.10:8080your server’s public address, port 8080 It waits five seconds, then:
You should seecurl: (28) Connection timed out after 5005 millisecondsThe number of milliseconds will be slightly different. The web server is running, but
quickcart-lab-sgallows nothing in, so your request was dropped and nothing replied.Allow port 8080 from your address: success
In the AWS console: EC2Security Groupsquickcart-lab-sgInbound rulesEdit inbound rules
Choose Add rule:
Type Port range Source Custom TCP 8080My IP My IP fills in your current public address with
/32on the end, meaning that one address only. Choose Save rules, then run the samecurlcommand again.Straight away, you should see an HTML page that includes<title>Directory listing for /</title>No restart, no waiting. The rule applied within seconds.
Allow a port nothing listens on: refused
Edit the inbound rules again and add one more:
Type Port range Source Custom TCP 8081My IP Save, then try port 8081. Use your address again:
Terminalcurl --max-time 5 http://203.0.113.10:8081Part What it means curl --max-time 5the same request, with the same 5-second limit :8081a port the security group now allows, but where no program is listening It fails at once, with a message likecurl: (7) Failed to connect to 203.0.113.10 port 8081 after 97 ms: Couldn't connect to serverOlder versions of curl say
Connection refusedinstead. Either way it’s fast: the packet got through the security group and the server itself answered “nothing here”. That’s a different problem from a time-out, with a different place to look.Clean-up step: Clean up
In the AWS console: EC2Instancesquickcart-labInstance stateTerminate (delete) instance
Confirm. Wait until its state is Terminated. A security group can’t be deleted while something is still using it.
In the AWS console: EC2Security Groups
Delete the groups in this order, one at a time: select, then Actions → Delete security groups.
Order Group Why this order 1 quickcart-lab-sgFree once the server is terminated 2 quickcart-db-sgNothing refers to it 3 quickcart-app-sgThe database group referred to it; now it’s gone 4 quickcart-alb-sgThe app group referred to it; now it’s gone If you try them in another order, AWS refuses because another group’s rule still names that group. Delete the one that refers to it first.
You should seeNone of the four
quickcart-groups in the list, and noquickcart-labinstance except as Terminated, which disappears on its own after about an hour.
Part 5: Check yourself
The app's security group allows port 8080 from the load balancer's security group. A request reaches the app. Which rule lets the app's reply go back to the load balancer?
How do you block one attacker's address with a security group?
A request through the load balancer fails with 504 Gateway Timeout after exactly 10 seconds. The app's logs show nothing at all. What do you check first?
Why should the database's rule say "from quickcart-app-sg" and not "from 10.0.11.0/24"?
New app servers boot but never become healthy. Session Manager can't connect to them. The system log shows the start-up script stuck on a package update. What's the most likely cause?
Part 6: Cheat sheet
| Term | In one line |
|---|---|
| Security group | A firewall on one resource. Allow rules only |
| Stateful | Replies to an allowed connection are allowed automatically |
| New group | No inbound rules; outbound allows all traffic |
| Source = another group | Allows whatever carries that group, at any address |
| The chain | ALB from 0.0.0.0/0 on 443 → app from ALB group on 8080 → database from app group on 5432 |
| Times out | Dropped by the network: security group, route table or network ACL |
| Refused, fast | Reached the server; nothing listening on that port |
| HTTP 500, fast | Reached the app; the code failed. Read its logs |
| 504 from the load balancer | It couldn’t reach a server within 10 seconds. Check the app’s security group |
| Can’t log in to a broken server? | Actions → Monitor and troubleshoot → Get system log |
| Network ACL | Subnet firewall. Stateless, numbered, allow and deny. Usually left at the default |