Alarms that actually fire

So an alarm has two jobs, and both have to be proved:

• Fire when something is really wrong — and only then.
• Reach a person who will act on it.

An alarm watches one metric, compares it with a threshold every period, and does something when the answer changes. Every CloudWatch alarm is in one of three states:

The five decisions in every alarm

Thresholds need reasons

A threshold picked because it’s a round number fires at the wrong times. Every threshold should answer “why this number?” These are the alarms QuickCart ended up with:

“M out of N”: how long before it fires

Two settings work together:

Asking for more datapoints makes the alarm slower, but calmer. When an alarm keeps switching between OK and ALARM, it’s flapping, and the fix is usually more datapoints or a better threshold.

Missing data is a decision too

Some metrics only report when something happens. The load balancer’s 5xx count, and a metric filter without a default value, publish nothing in a quiet minute. You choose what that silence means:

Getting it to a person: SNS

An alarm doesn’t send email itself. It publishes a message to an SNS topic, and the topic delivers it to everyone subscribed.

One topic can feed many alarms, and many subscribers. Change who gets alerts in one place.

Prove it fires

There are two ways to test an alarm, and they test different things:

Do both. The forced test is quick and safe. The real one is the only proof that the alarm would fire during an actual incident. In QuickCart’s build, removing the database’s security group rule made the app return 500s, and the error alarm fired within two minutes: that’s the evidence it works.

Alarm fatigue

When alarms fire and nothing needs doing, people learn to ignore them. Then they ignore the real one.

Composite alarms

When one thing breaks, several alarms often fire together: target 5xx, app errors and latency, all for one database problem. A composite alarm combines other alarms with a rule, and notifies once:

ALARM("quickcart-target-5xx") AND ALARM("quickcart-app-errors")

The individual alarms keep their states for the dashboard, but only the composite one sends the notification. Composite alarms cost a little more than a normal alarm. Start with a few well-tuned alarms and add composite ones when you see the same group firing together.

Here’s one metric — errors per minute — and an alarm on it. Pick what the metric does, then change the threshold, the datapoints and the missing-data setting. Watch the state row underneath, and count how many notifications would be sent.

What to notice: with 1 of 1, the one-minute blip wakes someone up. With 2 of 3 it doesn’t, and the real outage still fires, just a minute later. On the quiet night, the only thing that changes the result is what missing data means.

Pennies — one alarm (about $0.10 a month, charged by the hour, and the free tier covers ten), an SNS topic whose first 1,000 emails a month are free, and a few kilobytes of logs.

You’ll build an error alarm on a log metric, send it to your email, force it to fire, then make it fire for real. About 30 minutes.

You need the AWS CLI with a working login, and the console’s region set to N. Virginia (us-east-1).

1. Create the log group, stream and metric filter

   If you kept /quickcart/orders and its metric filter from Logs and metrics, skip to the next step.

   AWSIn the AWS console: CloudWatch→Log groups→Create log group

   Field	Enter
   Log group name	/quickcart/orders
   Retention setting	1 week

   Choose Create. Open the new group, and on Log streams choose Create log stream:

   Field	Enter
   Log stream name	lab-1

   Then on Metric filters, choose Create metric filter:

   Field	Enter
   Filter pattern	{ $.level = "ERROR" }
   Filter name	quickcart-app-errors
   Metric namespace	QuickCart/App
   Metric name	AppErrorCount
   Metric value	1
   Default value	0

   Choose Next between the pages, then Create metric filter.

   You should see

   The log group with one stream, lab-1, and one metric filter.

2. Create the SNS topic

   AWSIn the AWS console: Simple Notification Service→Topics→Create topic

   Field	Enter
   Type	Standard
   Name	quickcart-alerts

   Leave the rest and choose Create topic.

   You should see

   The topic’s page, with its ARN: arn:aws:sns:us-east-1:111122223333:quickcart-alerts, with your own account number.

3. Subscribe your email, and confirm it

   On the topic’s page, choose Create subscription.

   Field	Enter
   Topic ARN	already filled in
   Protocol	Email
   Endpoint	your own email address

   Choose Create subscription. Then open your email, find the message from AWS Notifications — it may be in spam — and choose Confirm subscription.

   You should see

   Back in SNS → Subscriptions, refresh: the status is Confirmed. If it still says Pending confirmation, nothing will reach you. Don’t go on until it says Confirmed.

4. Send some normal traffic, so the metric exists

   The console can only put an alarm on a metric that has data. Normal log lines make the metric filter publish its default value, 0.

   The console has no way to send log lines by hand, so this uses the CLI. First get the current time:

   TerminalCopy

   date +%s

   Part	What it means
   date	print the date and time
   +%s	as seconds since 1 January 1970

   You should see

   1790349700

   Yours will be larger. Add three zeros to turn it into milliseconds. Create normal.json, putting your number in place of the highlighted one:

   normal.jsonCopy

   [
     {"timestamp": 1790349700000, "message": "{\"level\":\"INFO\",\"route\":\"/orders\",\"status\":200,\"latency_ms\":41,\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790349700000, "message": "{\"level\":\"INFO\",\"route\":\"/orders\",\"status\":200,\"latency_ms\":38,\"instance_id\":\"i-0bb22\"}"}
   ]

   From the same folder, send it:

   TerminalCopy

   aws logs put-log-events --region us-east-1 --log-group-name /quickcart/orders --log-stream-name lab-1 --log-events file://normal.json

   Part	What it means
   aws logs put-log-events	send log events to CloudWatch Logs
   --region us-east-1	the region the log group is in
   --log-group-name /quickcart/orders	which log group
   --log-stream-name lab-1	which stream
   --log-events file://normal.json	read the events from this file

   You should see

   A short JSON reply, without rejectedLogEventsInfo. Wait two or three minutes before the next step, so the metric appears.

5. Create the alarm

   AWSIn the AWS console: CloudWatch→Alarms→All alarms→Create alarm

   Choose Select metric, then QuickCart/App → Metrics with no dimensions, tick AppErrorCount, and choose Select metric.

   Field	Enter
   Statistic	Sum
   Period	1 minute
   Threshold type	Static
   Whenever AppErrorCount is	Greater
   than	5

   Open Additional configuration:

   Field	Enter
   Datapoints to alarm	1 out of 1
   Missing data treatment	Treat missing data as good (not breaching threshold)

   Choose Next.

   Field	Enter
   Alarm state trigger	In alarm
   Send a notification to the following SNS topic	Select an existing SNS topic, then quickcart-alerts

   Choose Next.

   Field	Enter
   Alarm name	quickcart-app-errors
   Alarm description	More than 5 app errors in a minute. Runbook: https://example.com/runbooks/checkout-5xx

   Choose Next, check the summary, and choose Create alarm.

   You should see

   quickcart-app-errors in the list. It starts as Insufficient data, and within a minute or two changes to OK.

6. Force it to fire

   The console has no way to set an alarm’s state, so this uses the CLI:

   TerminalCopy

   aws cloudwatch set-alarm-state --region us-east-1 --alarm-name quickcart-app-errors --state-value ALARM --state-reason "Testing the notification path"

   Part	What it means
   aws cloudwatch set-alarm-state	change an alarm’s state by hand, for testing
   --region us-east-1	where the alarm is
   --alarm-name quickcart-app-errors	which alarm
   --state-value ALARM	the state to put it in
   --state-reason "Testing the notification path"	a note that shows in the alarm’s history, so others know it was a test

   It prints nothing when it works.

   You should see

   Within a minute, an email with the subject ALARM: "quickcart-app-errors" in US East (N. Virginia). In the console the alarm shows In alarm, then goes back to OK at its next evaluation, about a minute later, because the metric itself is fine.

7. Make it fire for real

   Now make the metric itself cross the threshold: eight error lines in one minute is more than 5. Run date +%s again for a fresh time — the alarm only looks at the last minute or so, so the timestamp must be now. Create errors.json with your new number:

   errors.jsonCopy

   [
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0bb22\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0bb22\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0bb22\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790350000000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0bb22\"}"}
   ]

   Send it straight away:

   TerminalCopy

   aws logs put-log-events --region us-east-1 --log-group-name /quickcart/orders --log-stream-name lab-1 --log-events file://errors.json

   Part	What it means
   aws logs put-log-events	send log events to CloudWatch Logs
   --region us-east-1	the region the log group is in
   --log-group-name /quickcart/orders	which log group
   --log-stream-name lab-1	which stream
   --log-events file://errors.json	read the eight error lines from this file

   You should see

   Within about two or three minutes, the alarm goes to In alarm and a second email arrives. This time the email’s reason names the real datapoint — something like Threshold Crossed: 1 out of the last 1 datapoints [8.0 (25/09/26 15:26:00)] was greater than the threshold (5.0), with your own time. A few minutes later, with no more errors, the alarm returns to OK.

   If nothing happens after five minutes, the timestamp was probably too old by the time you sent it. Run date +%s, update the file and send it again.

8. Read the alarm's history

   AWSIn the AWS console: CloudWatch→Alarms→All alarms→quickcart-app-errors→History

   You should see

   Each state change with its time and reason: the forced one says Testing the notification path; the real one names the datapoint that crossed the threshold. Each Action row shows the notification being sent to quickcart-alerts. This history is your evidence the alarm works.

9. Clean-up step: Clean up

   Delete the alarm:

   AWSIn the AWS console: CloudWatch→Alarms→All alarms→quickcart-app-errors→Actions→Delete

   Delete the topic, which removes its subscription too:

   AWSIn the AWS console: Simple Notification Service→Topics→quickcart-alerts→Delete

   Type delete me when asked, and choose Delete. Then delete the log group:

   AWSIn the AWS console: CloudWatch→Log groups→/quickcart/orders→Actions→Delete log group(s)

   You should see

   No alarms, no quickcart-alerts topic and no /quickcart/orders log group left. The subscription may linger for a short time in the list as it’s removed.