Who changed what

Every change in AWS leaves a record. This lesson is about finding it.

Everything you do in AWS, whether you click in the console, run a CLI command, or let Terraform apply, becomes a call to AWS’s API. CloudTrail records those calls: who made each one, what it was, when, from where, and whether it worked.

What gets recorded

Failed calls are recorded too. An event with "errorCode": "AccessDenied" shows exactly which action someone tried and was refused — which makes CloudTrail the best place to debug a permissions problem, as the IAM lesson shows.

Event history: free, on, 90 days

Every account has event history turned on from the start. It keeps 90 days of management events, costs nothing, and can’t be turned off. You open it in the console under CloudTrail → Event history.

You filter by one lookup attribute:

Most events are someone looking at something: Describe…, List…, Get…. Filtering by Read only = false removes all of them and leaves only the changes. That’s usually the first filter to try when asking “what changed?”

Reading an event record

Each event has a record in JSON. These are the fields that answer “who did what”:

userIdentity has a type that tells you what sort of caller it was:

Trails: keeping it longer, and in one place

Event history is enough to answer “who did this, this week?” For anything more, create a trail: it writes every event to files in an S3 bucket, where you decide how long they stay.

AWSIn the AWS console: CloudTrail→Trails→Create trail

Trails created in the console record every region, which is what you want. Costs, roughly and as of writing: the first copy of management events is free, apart from the S3 storage; data events cost about $0.10 per 100,000 events; a second copy of management events is charged too. The bucket holds your audit record, so lock it down: no public access, and few people allowed to delete from it.

CloudTrail Lake is a paid option that stores events for years and lets you search them with SQL. You won’t need it for QuickCart; if you do later, check the current pricing and availability first. A trail in S3 plus Athena can answer the same questions.

Someone removed the rule that lets the app reach the database. Below is the last hour of QuickCart’s event history. Try each lookup, then open the event that answers the question. Finally, try to find who deleted a file in S3.

What to notice: unfiltered, most of the history is people looking at things. Read-only: false or the exact event name gets you to the one change in seconds. And the S3 file deletion isn’t there at all: that’s a data event, and event history doesn’t record them.

Free — security groups cost nothing, and event history is free.

You’ll make a change, undo part of it, and then find both in CloudTrail — exactly as you would after an incident. About 20 minutes, most of it waiting for events to arrive.

Check the region selector says N. Virginia (us-east-1). Event history shows one region at a time, so the change and the lookup must be in the same region.

1. Create a security group with one rule

   AWSIn the AWS console: EC2→Network & Security→Security Groups→Create security group

   Field	Enter
   Security group name	quickcart-trail-lab
   Description	CloudTrail lab
   VPC	the default VPC, or any VPC you have

   Under Inbound rules, choose Add rule:

   Type	Port range	Source
   Custom TCP	8080	Custom, 10.0.0.0/16

   Leave the outbound rules as they are, and choose Create security group.

   You should see

   The new group’s page. Copy its Security group ID, which starts sg-. You’ll look it up later.

2. Remove the rule, as someone might during an incident

   On the group’s page, open the Inbound rules tab and choose Edit inbound rules. Choose Delete next to the 8080 rule, then Save rules.

   You should see

   Inbound rules (0). Note the time. Now wait about 5 minutes: events usually take a few minutes to reach CloudTrail.

3. Find the change by its event name

   AWSIn the AWS console: CloudTrail→Event history

   Field	Enter
   Lookup attributes	Event name
   Value	RevokeSecurityGroupIngress

   You should see

   One row (or more, if you’ve done this before): Event name RevokeSecurityGroupIngress, User name with your own sign-in name, Event source ec2.amazonaws.com, and your security group ID under Resource name.

   Choose the event name to open it.

   You should see

   The event’s details, and below them the Event record: the full JSON. Find these fields in it:

   • eventName: RevokeSecurityGroupIngress
   • userIdentity: you. type is IAMUser if you signed in as an IAM user, or AssumedRole if you signed in through single sign-on
   • sourceIPAddress: your own internet address
   • requestParameters: your group’s ID, and the rule that was removed — shown either as the port and range, or as a rule ID starting sgr-
   • readOnly: false

4. See the whole life of one resource

   Change the lookup:

   Field	Enter
   Lookup attributes	Resource name
   Value	your security group ID, like sg-0db5e1f2

   You should see

   Every change to that one group, newest first: RevokeSecurityGroupIngress, then AuthorizeSecurityGroupIngress (the rule being added), then CreateSecurityGroup. The console did it in two calls; you clicked once. Your list may also include read-only calls the console made to show you the group.

5. Ask what changed in the last hour

   This is the question to ask early in any incident.

   Field	Enter
   Lookup attributes	Read only
   Value	false

   Set the time range at the top to the last hour.

   You should see

   Only changes: your three security group calls, plus anything else that changed in this region in the last hour — including changes made by AWS services for you, such as auto scaling launching servers, if you have any running.

6. Clean-up step: Clean up

   AWSIn the AWS console: EC2→Network & Security→Security Groups→quickcart-trail-lab→Actions→Delete security groups

   Choose Delete.

   You should see

   The group is gone. If you look in event history in a few minutes, you’ll find DeleteSecurityGroup with your name on it. There’s nothing else to clean up: event history can’t be turned off, and it costs nothing.