Runbooks

A runbook turns “someone who knows” into “anyone on call”. This lesson is about writing one that works when it’s needed, and proving that it does.

A runbook is a written procedure for one specific situation, detailed enough that anyone on the team could follow it at 3am, half awake, without asking anyone.

It is not documentation of how the system works. It’s a checklist for one job: “checkout returns 5xx”, “the database disk is nearly full”, “roll back a release”.

The parts of a good runbook

Exact steps, not good intentions

Every exact step names where, what, and what to do with the answer. Each step that makes a change also says how to undo it.

Test it, and time it

A runbook nobody has run is a guess, like an alarm nobody has seen fire. Test it with a dry run:

After each test: fix every gap you found, write down the time it took, and update Last tested. If it took 37 minutes and your target is 15, the runbook isn’t finished.

Keep them where people will find them

• Link each runbook from its alarm. Put the link in the alarm’s description; it’s included in the notification, so the first thing the on-call person reads contains the first step.
• Keep runbooks in the repository, as Markdown next to the code. Changes are reviewed like code, and the history shows who changed what.
• Automate the steps that never change. A rollback that is always the same five commands should be one pipeline job. The runbook then says “run this job”, and the job can’t mistype a value. Canary from a pipeline builds one.

A complete example: checkout returns 5xx

This is QuickCart’s runbook for its most important alarm. Use it as a model.

Runbook: checkout returns 5xx

Step 1: confirm it’s real (2 minutes). Open the quickcart-checkout dashboard. If the 5xx rate is above 1%, it’s real: carry on. If it’s zero and the alarm has already returned to OK, note it in the alarm review and stop.

Step 2: is the load balancer happy with the servers? (1 minute).

AWSIn the AWS console: EC2→Load Balancing→Target Groups→quickcart-checkout-tg→Targets

Step 3: what are the errors? (2 minutes).

AWSIn the AWS console: CloudWatch→Logs Insights

Choose the log group /quickcart/orders, set the range to 30m, and run:

filter level = "ERROR"
| stats count() as errors, avg(latency_ms) as avg_ms by error_type
| sort errors desc

Step 4: what changed? (2 minutes). Look at the last two runs of the deploy pipeline. Then open CloudTrail → Event history with Read only set to false for the last hour. If a deploy went out in the hour before the alarm, go to step 5.

Step 5: roll back (3 minutes). Run the quickcart-rollback job with VERSION set to the version before the latest, shown on the job’s last successful run. The on-call role has permission to run it.

• To undo this step: run the same job with the newer version. Only do this once the cause is fixed.

Step 6: verify (5 minutes). It’s fixed when both are true:

• the 5xx alarm is back to OK, and
• the 5xx rate on the dashboard stays under 1% for 5 minutes in a row.

Then post in the incident channel: what happened, what you did, and the times. A postmortem follows within two working days.

Below is the checkout runbook as it was first written: from memory, never tested. Run it one step at a time and watch the clock. Then switch to the version written after the dry run and run it again.

What to notice: the steps are the same six in both versions. What changed is that each one got exact names, a link that works, a saved query, a tested permission and a clear finish line. None of those gaps was visible until someone ran the runbook with a clock.

Free — you’ll write a document and time yourself. Nothing is created in AWS.

You’ll write a runbook for one QuickCart alarm, then dry-run it with a stopwatch and fix what you find. About 30 minutes. You don’t need anything running: a read-through dry run works on paper, and it finds most gaps.

1. Pick one alarm

   Choose one alarm from Alarms that actually fire, other than the 5xx one, which has an example above. Good choices:

   Alarm	Situation
   quickcart-unhealthy-hosts	A server is failing its health checks
   quickcart-db-connections	The database is running out of connections
   quickcart-db-storage	The database disk is nearly full

2. Write it from the template

   Create a file called runbooks/ followed by a short name, such as runbooks/db-storage.md, and copy this template into it:

   runbooks/db-storage.mdCopy

   # Runbook: <what is happening>
   
   | Field | Value |
   |---|---|
   | Trigger | <the alarm name, and its condition> |
   | Impact | <what customers experience> |
   | Owner | <the team> |
   | Target time | <restored within N minutes of the alarm> |
   | Last tested | never |
   | Escalation | <who to call, and after how long> |
   
   ## Step 1: confirm it's real (<N> minutes)
   <exactly what to open, and what number means it's real>
   
   ## Step 2: <name> (<N> minutes)
   <where to go, what to run, what to look for>
   <what each possible answer means, and which step to go to>
   
   ## Step 3: fix (<N> minutes)
   <the exact action>
   To undo this step: <the exact way back>
   
   ## Step 4: verify (<N> minutes)
   Fixed when: <a measurable test, and for how long>

   Replace everything in angle brackets. For each step, write how many minutes you think it will take.

   You should see

   A runbook of four to six steps. Every step names a place, an action and what to do with the answer. Every step that changes something has a way back.

3. Read it through as a stranger

   Read each step aloud, and for each one ask:

   Question	If the answer is no
   Would someone who’s never seen this system know exactly where to click?	Add the console path or the link
   Does it say what to do with each possible answer?	Add the branches
   Could the on-call person do it with their own permissions?	Note the permission needed, and check it
   Is “fixed” something you can measure?	Replace it with a number and a duration

   Mark every gap you find.

4. Dry-run it with a stopwatch

   Start a stopwatch on your phone. Pretend the alarm has just fired, and follow your runbook step by step. If you have the environment from A self-healing environment or a staging copy, do the steps for real. If not, open each console page the step names and find each thing it mentions, without changing anything.

   Write down how long each step really took next to your estimate.

   You should see

   At least one step took much longer than you expected. That’s normal on a first run, and it’s exactly what the dry run is for.

5. Fix the gaps and record the test

   For every step that was slow or unclear, make it exact. Then update the header:

   Field	Enter
   Last tested	today’s date, dry run or read-through, and the total time

   If the total is over your target time, run it again after the fixes.

6. Link it from the alarm

   When the alarm exists, put the runbook’s link in its description:

   AWSIn the AWS console: CloudWatch→Alarms→All alarms→the alarm→Actions→Edit

   Choose Next until you reach Add name and description.

   Field	Enter
   Alarm description	One line on what it means, then Runbook: and the link, such as https://example.com/runbooks/db-storage

   You should see

   The link in the alarm’s details. The next notification will include it.

   If you deleted your alarms after the last lesson, keep the description ready for when you build them again in the projects.

7. Clean-up step: Clean up

   Nothing was created in AWS, so there’s nothing to delete. Keep your runbook file: put it in the repository next to your QuickCart code, where you’ll use it in A self-healing environment.

   You should see

   One tested runbook, with a real date in Last tested.