Real incidents

Incidents are expensive. The only way to get something back for the cost is to learn from them — which only works if people can say what really happened.

This course’s projects were built for real, and things broke along the way. Every failure below actually happened during those builds. Names are left out; the details are real. Almost none of them were code: they were a missing rule, a port number, a typo. That’s where most outages come from.

What went wrong while building the course

1. New servers never became healthy

2. The same error, two different causes

3. The health check that kept the site up

4. The alarm test that proved the whole chain

5. Container tasks restarting, “Request timed out”

6. The load balancer wouldn’t accept the default VPC

7. The image scan said “not found”

8. The load test said every request failed

9. A placeholder in the manifest

10. The Kubernetes console said “Unauthorized”

Writing a blameless postmortem

A postmortem is a short written account of an incident: what happened, why, and what will change. It’s written within a few days, while memories are fresh, and shared widely.

Blameless means it assumes everyone acted sensibly with what they knew at the time, and asks what made the mistake possible. Not because nobody is responsible, but because blame stops people telling you what really happened — as Leo showed.

The sections of a postmortem

A simple way to get from the direct cause to the conditions behind it is to keep asking why? “The servers couldn’t boot.” Why? “They couldn’t make outbound connections.” Why? “The security group had no outbound rules.” Why? “It was made by hand, and nothing checks it.” Stop when you reach something you can change.

Replay three of the incidents above, one event at a time. Watch which phase each event belongs to, and notice the dead ends.

What to notice: every replay has at least one moment where the obvious next step led nowhere. That’s normal, and a good postmortem timeline includes it: the dead ends show where better tools, logs or runbooks would have saved time.

Free — you’ll write a document. Nothing is created in AWS.

You’ll write a blameless postmortem for incident 1, new servers never became healthy, using a template. About 30 minutes.

1. Copy the template

   Create postmortems/servers-never-healthy.md and copy this in:

   postmortems/servers-never-healthy.mdCopy

   # Postmortem: <title>
   
   Status: draft
   Date of incident: <date>
   Author: <you>
   
   ## Summary
   <Two or three sentences: what broke, for how long, and what fixed it.>
   
   ## Impact
   <Who was affected, how, and for how long. Use numbers.>
   
   ## Timeline (UTC)
   | Time | What happened |
   |---|---|
   | <hh:mm> | <first symptom> |
   | <hh:mm> | <alarm or report> |
   | <hh:mm> | <each investigation step, including dead ends> |
   | <hh:mm> | <cause found> |
   | <hh:mm> | <fix applied> |
   | <hh:mm> | <recovery confirmed> |
   
   ## Detection
   <How did we find out? How could we have found out sooner?>
   
   ## Causes
   Direct cause: <the one thing that broke>
   Contributing conditions:
   - <what made it possible>
   - <what made it slow to find>
   
   ## What went well
   - <keep doing>
   
   ## What went badly
   - <slowed us down>
   
   ## Where we got lucky
   - <could have been worse>
   
   ## Action items
   | Action | Owner | Due |
   |---|---|---|
   | <specific change to the system> | <name> | <date> |

2. Fill in the timeline

   Use the replay above: choose Servers never healthy and step through it. Copy each event into the timeline, including the dead end.

   You should see

   Six rows, from the first failed health check to the servers becoming healthy, with the Session Manager dead end in the middle.

3. Write the summary and impact

   For impact, imagine this happened in production at 10:00: say what customers experienced and for how long. If there were other healthy servers still serving, say so — “no customer impact, because the existing servers kept serving” is a real and useful impact statement.

4. Find the causes, with five whys

   Start from “the new servers never became healthy” and ask why? until you reach something you can change. Write the direct cause, then at least two contributing conditions.

   You should see

   Something like: the direct cause is the missing outbound rule. Contributing conditions: the security group was made by hand with nothing to check it; Session Manager depends on the same outbound access, so the usual way in failed too; nobody on the team knew about the system log.

5. Write action items that change the system

   Write at least three. Each must be specific, have an owner and a due date, and change the system rather than asking people to be more careful. Some ideas:

   Action	Why it helps
   Define the security group in Terraform, reviewed in a pull request	A missing rule is visible in review
   Add “Get system log” to the unhealthy-hosts runbook, as step 2	The next person knows where to look
   Alarm when the auto scaling group launches more than N servers in 15 minutes	A replacement loop gets noticed quickly

6. Do a blameless pass

   Read the whole document looking for any sentence that names a person as the cause. Rewrite each one to say what made the mistake possible, as in the table in the idea. Change Status to ready for review.

   You should see

   No sentence says who got it wrong. Every cause points at something that can be changed.

7. Clean-up step: Clean up

   Nothing was created in AWS, so there’s nothing to delete. Keep the postmortem in your repository, next to your runbooks. A folder of postmortems is how a team remembers what it has learned.

   You should see

   One postmortem, ready for review, with at least three action items that each have an owner and a date.