Laptop to Production

Security groups

The firewall on every resource, and chaining them.

Time
40 minutes
Level
Beginner

Part 1: The problem

QuickCart, the night the servers kept dying

Priya tidied up QuickCart’s security groups one evening. On quickcart-app-sg she removed the outbound rule. The app only ever answered requests, she reasoned, so why would it need to send anything out?

Nothing broke straight away. The running servers kept serving orders. But the next morning the auto scaling group started new servers, and none of them came up.

The auto scaling group launched a server, waited, marked it unhealthy, terminated it, and launched another. All morning. Session Manager couldn’t connect to any of them. The load balancer said they failed their health checks. Nobody could see why.

Leo finally opened the console log of one doomed server with Actions → Monitor and troubleshoot → Get system log. The start-up script was stuck on dnf update. It wasn’t failing, just waiting. So was the Session Manager agent. Neither could reach the internet, because a server with no outbound rule can’t start any connection at all.

The app didn’t need to send anything to customers. But a server does plenty of things on its own: it downloads updates, installs the app, and reports in to AWS. Every one of those is a connection the server starts.

This lesson is about getting those rules right: what a security group allows, what it quietly drops, and how to tell a network problem from a code problem in seconds.

Part 2: The idea

A security group is a firewall attached to one resource: a server, a load balancer, a database. It’s a list of rules saying what may connect to that resource, and what the resource may connect to.

The route table decides whether traffic can find a path to a subnet. The security group decides whether the resource lets it in. Both have to say yes.

Four rules about how security groups behave

BehaviourWhat it meansWhy it matters
Allow onlyEvery rule allows something. There are no deny rulesAnything not allowed is blocked. You never write “block this”
StatefulOnce a connection is allowed in, its replies are allowed out, automatically. The same is true the other way roundYou only write rules for connections that start. Replies take care of themselves
SilentBlocked traffic is dropped without a reply. The sender hears nothingThe sender waits until it gives up. This is why blocked traffic shows up as a time-out
ImmediateA rule change applies within seconds, to every resource using the groupNo restart. Also no undo button, so change rules carefully

What’s in a rule

Each rule has three parts:

PartExampleWhat it means
ProtocolTCPThe kind of traffic. Nearly everything you’ll write is TCP
Port8080Which door on the resource. A web server might listen on 443, an app on 8080, PostgreSQL on 5432
Source (inbound) or destination (outbound)quickcart-alb-sgWho may connect. An address range such as 0.0.0.0/0, or another security group

What a new security group allows

DirectionA new security group starts withSo
InboundNo rulesNothing can connect in until you add a rule
OutboundOne rule: all traffic to 0.0.0.0/0The resource can start connections to anywhere

That outbound rule is the one Priya deleted. It looks unnecessary, which is why people remove it.

Name the group before, not an address range

A source can be another security group. The rule then means anything that has that security group attached, wherever it is and whatever its address.

That’s how QuickCart chains its three tiers, each one letting in only the one in front of it:

Security groupAttached toInbound ruleWhy
quickcart-alb-sgThe load balancerHTTPS, port 443, from 0.0.0.0/0Customers can be anywhere
quickcart-app-sgThe app serversTCP, port 8080, from quickcart-alb-sgOnly the load balancer may reach the app
quickcart-db-sgThe databasePostgreSQL, port 5432, from quickcart-app-sgOnly the app may reach the database

If the auto scaling group adds ten servers, the database rule needs no change: the new servers carry quickcart-app-sg, so they’re allowed. And if someone puts a test server in the app subnet without that group, it can’t reach the database, even though its address is in the same range.

Key point: A request that times out was dropped by the network. A request that fails fast reached something that answered.

Time-out or error: the fastest diagnosis you’ll ever make

What you seeWhat happenedWhere to look
Waits, then times outPackets were silently dropped. Nothing answeredSecurity groups, route tables, network ACLs
Connection refused, straight awayThe packet arrived, but nothing is listening on that portIs the app running? Is it listening on that port?
An HTTP error such as 500, straight awayThe app received the request and failedThe app’s logs

Behind a load balancer the same rule holds. The load balancer tries to connect to a server for 10 seconds, then returns 504 Gateway Timeout. A 504 with nothing in the app’s logs almost always means a security group.

Network ACLs, briefly

A VPC has a second, older firewall: the network ACL. You’ll rarely change it, but you should recognise it.

Security groupNetwork ACL
Attached toOne resourceA whole subnet
RulesAllow onlyAllow and deny
OrderNo order. All rules are checkedNumbered. The lowest-numbered match wins
RepliesStateful: replies are automaticStateless: replies need their own rule
DefaultNew group: nothing in, everything outThe VPC’s default ACL allows everything both ways

Because a network ACL is stateless, a reply needs its own rule. Replies go back to the random high port the other side picked, called an ephemeral port, usually somewhere between 1024 and 65535. Forgetting that return rule is the classic network ACL mistake. QuickCart leaves the default network ACL alone and does all its filtering with security groups.

Part 3: See it

Below is QuickCart’s chain: load balancer, app, database, each with its own security group. Send a request with every rule right. Then break one rule at a time, send again, and watch the clock at the bottom. Finally, try a bug in the code instead.

What's wrong
A customer's phone sends a request to the load balancer, which forwards it to an app server, which queries the database. Each of the three has its own security group with its own rules.Customerthe QuickCart appquickcart-alb-sgLoad balancerpublic subnetin: 443 from 0.0.0.0/0out: all trafficquickcart-app-sgApp serverlistens on 8080in: 8080 from alb-sgout: all trafficquickcart-db-sgDatabaselistens on 5432in: 5432 from app-sgout: all trafficCustomer has waited0.00 sWhat the customer getsnothing sent yet
ReadyEvery rule is right. Choose Send a request to follow an order through the three security groups. Then break a rule and send another.

What to notice: every broken rule gave the same result: a long wait, then a time-out, and nothing in the app’s logs about why. The bug in the code failed in a fraction of a second. How long a failure takes tells you where to look.

Part 4: Do it

Pennies — security groups are free. The test server costs about two cents an hour while it runs, including its public IP address (prices change). You delete it at the end.

You’ll build QuickCart’s three-group chain, then launch one small server and see a time-out, a success and a refused connection with your own eyes. About 30 minutes.

Check the region selector at the top right says N. Virginia (us-east-1). The lab uses your account’s default VPC, which AWS creates in every region. If you deleted it, create the VPC from the VPC lesson and choose it instead, with a public subnet.

  1. Create the load balancer's security group

    In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group

    FieldEnter
    Security group namequickcart-alb-sg
    DescriptionQuickCart load balancer: HTTPS from anywhere
    VPCyour default VPC

    Under Inbound rules, choose Add rule:

    TypeSource
    HTTPS (port 443 fills in)Anywhere-IPv4 (0.0.0.0/0)

    Leave Outbound rules as they are: one rule, All traffic to 0.0.0.0/0. Choose Create security group.

    You should see

    The group’s details page, with one inbound rule and one outbound rule.

  2. Create the app's security group, pointing at the first

    In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group

    FieldEnter
    Security group namequickcart-app-sg
    DescriptionQuickCart app servers: 8080 from the load balancer only
    VPCyour default VPC

    Under Inbound rules, choose Add rule:

    TypePort rangeSource
    Custom TCP8080Custom, then start typing quickcart-alb-sg and pick it from the list

    Leave the outbound rule. Choose Create security group.

    You should see

    The inbound rule’s source shows the ID of quickcart-alb-sg, starting sg-, instead of an address range.

  3. Create the database's security group, pointing at the app

    In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group

    FieldEnter
    Security group namequickcart-db-sg
    DescriptionQuickCart database: PostgreSQL from the app only
    VPCyour default VPC
    TypeSource
    PostgreSQL (port 5432 fills in)Custom, then pick quickcart-app-sg

    Choose Create security group. That’s the whole chain: three groups, three inbound rules, and no address ranges except the front door.

  4. Create a test group with no inbound rules

    Now you’ll see the three outcomes for yourself. Create one more group:

    In the AWS console: EC2Network & SecuritySecurity GroupsCreate security group

    FieldEnter
    Security group namequickcart-lab-sg
    DescriptionLab: test time-outs from my laptop
    VPCyour default VPC

    Add no inbound rules. Choose Create security group.

  5. Launch a small server running a web server on port 8080

    In the AWS console: EC2InstancesLaunch instances

    FieldEnter
    Namequickcart-lab
    Application and OS ImagesAmazon Linux 2023 AMI (the default)
    Instance typet3.micro
    Key pairProceed without a key pair
    Network settings → Edit → Auto-assign public IPEnable
    Firewall (security groups)Select existing security group, then quickcart-lab-sg

    Open Advanced details. In the User data box at the bottom, paste this script. It runs once, as the server first starts:

    user data
    #!/bin/bash
    systemd-run --unit quickcart-web /usr/bin/python3 -m http.server 8080
    PartWhat it means
    #!/bin/bashrun this script with bash
    systemd-runstart a program as a background service, so it keeps running after the script ends
    --unit quickcart-webthe service’s name
    /usr/bin/python3 -m http.serverPython’s built-in web server. Amazon Linux 2023 includes Python
    8080the port it listens on, like QuickCart’s app

    Choose Launch instance. Wait until the instance shows Running and its status check shows 2/2 checks passed, about two minutes. Then select it and copy its Public IPv4 address.

  6. Try it with no rule: a time-out

    On your computer, open Terminal and run the command below. Replace the highlighted address with the public IPv4 address you copied:

    Terminal
    curl --max-time 5 http://203.0.113.10:8080
    PartWhat it means
    curlmake a web request from the terminal and print the answer
    --max-time 5give up after 5 seconds instead of waiting for minutes
    http://203.0.113.10:8080your server’s public address, port 8080

    It waits five seconds, then:

    You should see
    curl: (28) Connection timed out after 5005 milliseconds

    The number of milliseconds will be slightly different. The web server is running, but quickcart-lab-sg allows nothing in, so your request was dropped and nothing replied.

  7. Allow port 8080 from your address: success

    In the AWS console: EC2Security Groupsquickcart-lab-sgInbound rulesEdit inbound rules

    Choose Add rule:

    TypePort rangeSource
    Custom TCP8080My IP

    My IP fills in your current public address with /32 on the end, meaning that one address only. Choose Save rules, then run the same curl command again.

    Straight away, you should see an HTML page that includes
    <title>Directory listing for /</title>

    No restart, no waiting. The rule applied within seconds.

  8. Allow a port nothing listens on: refused

    Edit the inbound rules again and add one more:

    TypePort rangeSource
    Custom TCP8081My IP

    Save, then try port 8081. Use your address again:

    Terminal
    curl --max-time 5 http://203.0.113.10:8081
    PartWhat it means
    curl --max-time 5the same request, with the same 5-second limit
    :8081a port the security group now allows, but where no program is listening
    It fails at once, with a message like
    curl: (7) Failed to connect to 203.0.113.10 port 8081 after 97 ms: Couldn't connect to server

    Older versions of curl say Connection refused instead. Either way it’s fast: the packet got through the security group and the server itself answered “nothing here”. That’s a different problem from a time-out, with a different place to look.

  9. Clean-up step: Clean up

    In the AWS console: EC2Instancesquickcart-labInstance stateTerminate (delete) instance

    Confirm. Wait until its state is Terminated. A security group can’t be deleted while something is still using it.

    In the AWS console: EC2Security Groups

    Delete the groups in this order, one at a time: select, then Actions → Delete security groups.

    OrderGroupWhy this order
    1quickcart-lab-sgFree once the server is terminated
    2quickcart-db-sgNothing refers to it
    3quickcart-app-sgThe database group referred to it; now it’s gone
    4quickcart-alb-sgThe app group referred to it; now it’s gone

    If you try them in another order, AWS refuses because another group’s rule still names that group. Delete the one that refers to it first.

    You should see

    None of the four quickcart- groups in the list, and no quickcart-lab instance except as Terminated, which disappears on its own after about an hour.

Part 5: Check yourself

The app's security group allows port 8080 from the load balancer's security group. A request reaches the app. Which rule lets the app's reply go back to the load balancer?

How do you block one attacker's address with a security group?

A request through the load balancer fails with 504 Gateway Timeout after exactly 10 seconds. The app's logs show nothing at all. What do you check first?

Why should the database's rule say "from quickcart-app-sg" and not "from 10.0.11.0/24"?

New app servers boot but never become healthy. Session Manager can't connect to them. The system log shows the start-up script stuck on a package update. What's the most likely cause?

Part 6: Cheat sheet

On one card
TermIn one line
Security groupA firewall on one resource. Allow rules only
StatefulReplies to an allowed connection are allowed automatically
New groupNo inbound rules; outbound allows all traffic
Source = another groupAllows whatever carries that group, at any address
The chainALB from 0.0.0.0/0 on 443 → app from ALB group on 8080 → database from app group on 5432
Times outDropped by the network: security group, route table or network ACL
Refused, fastReached the server; nothing listening on that port
HTTP 500, fastReached the app; the code failed. Read its logs
504 from the load balancerIt couldn’t reach a server within 10 seconds. Check the app’s security group
Can’t log in to a broken server?Actions → Monitor and troubleshoot → Get system log
Network ACLSubnet firewall. Stateless, numbered, allow and deny. Usually left at the default

Finished the build and the quiz? Mark it done to fill in its stop on the route map.