Logs and metrics

QuickCart needs two things:

• Logs that leave the server the moment they’re written, so they survive the server.
• Logs that can answer questions: filter, count, group, average.

This lesson covers both, and metrics: the numbers you’ll hang alarms on in the next lesson.

A running system tells you what it’s doing in two ways.

In AWS, both live in CloudWatch.

Where logs go: CloudWatch Logs

Retention defaults to “Never expire”. Every log group you create keeps everything forever unless you choose otherwise. Choose a period every time.

Getting logs off the server

The app writes to a file, as it always has. Something has to copy each line to CloudWatch as it’s written.

The agent reads a JSON config file. This one follows the app’s log file and sends it to /quickcart/orders, one stream per server:

{
  "agent": {
    "metrics_collection_interval": 60
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/app/app.log",
            "log_group_name": "/quickcart/orders",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 30,
            "timezone": "UTC"
          }
        ]
      }
    }
  },
  "metrics": {
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent"]
      },
      "disk": {
        "measurement": ["used_percent"],
        "resources": ["/"]
      }
    }
  }
}

The server also needs an IAM role that lets it write to CloudWatch; AWS’s managed policy CloudWatchAgentServerPolicy covers it. On the server, this starts the agent with that file:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/quickcart-agent.json

Why the format of each line matters

Here is the same failure, written two ways:

2026-09-25T02:06:15Z ERROR GET /orders 500 database_unavailable

{"level":"ERROR","route":"/orders","status":500,"latency_ms":3004,"error_type":"OperationalError","instance_id":"i-0aa11","detail":"Can't connect to MySQL server on 'quickcart-db' (timed out)"}

The plain line is easy to read and impossible to query. To count errors by type, you’d have to guess at words inside the text. The JSON line is a set of named fields. CloudWatch finds them automatically, so you can ask “how many errors of each error_type?” or “what was the average latency_ms?” This is called structured logging.

A few habits make structured logs useful:

Metrics: numbers over time

A metric is one number, reported again and again. AWS services publish many for free. Five words describe any metric:

The statistic matters. Counts are added up with Sum: errors per minute. Levels are averaged: CPU with Average. Response times use a percentile: p95 is the time 95% of requests beat, which shows the slow minority an average hides.

A few metrics QuickCart will use constantly:

EC2 reports every 5 minutes unless you turn on detailed monitoring, which reports every minute. The difference matters for alarms: a problem can be 5 minutes old before a 5-minute metric even shows it.

Metric filters: turning log lines into a metric

A metric filter watches a log group and adds to a metric whenever a line matches a pattern. It’s how “an ERROR line was written” becomes a number you can alarm on.

The { $.field = value } pattern only works on JSON lines — one more reason to log structured. Metric filters only see lines that arrive after they’re created. They never go back over old logs.

Logs Insights: asking questions of your logs

Logs Insights is a query tool in the CloudWatch console. You pick log groups and a time range, write a query, and it scans the logs. Each query is a series of commands joined by |, like a pipe in the terminal: each step works on what the step before it produced.

Fields that start with @ are added by CloudWatch: @timestamp is the event’s time, and @message is the whole raw line. Fields from your JSON appear under their own names.

filter level = "ERROR"
| stats count() as errors by bin(1m)
| sort @timestamp asc

That one query shows when the trouble started, when it stopped and how bad it got. It’s usually the first thing to run in an incident.

Below are twelve log lines from QuickCart’s orders API: a few normal requests and six errors. Switch between plain text and structured JSON, then try to find out what went wrong. Filter by level and error_type, and count by type or by server.

fields @timestamp, level, error_type, latency_ms, instance_id
| sort @timestamp asc

What to notice: in plain text, all six errors look identical, and no query can split them. In JSON, one stats line reveals two different problems: one failing in 5 ms and one timing out at 3,000 ms. The customer saw the same message for both.

Pennies — a few kilobytes of logs, a few Logs Insights queries and one custom metric. Well under a cent, and the free tier covers most of it.

You’ll create a log group, send it structured log lines, turn the errors into a metric, ask the logs questions with Logs Insights, and set retention. About 30 minutes.

You need the AWS CLI with a working login in your terminal. Set up your computer covers it. Check the region selector in the console says N. Virginia (us-east-1).

1. Create the log group

   AWSIn the AWS console: CloudWatch→Log groups→Create log group

   In newer versions of the console, Log groups sits under Logs, and may be called Log Management.

   Field	Enter
   Log group name	/quickcart/orders
   Retention setting	Never expire — leave the default for now. You’ll change it in the last step
   Log class	Standard

   Leave everything else empty and choose Create.

   You should see

   /quickcart/orders in the list, with Retention: Never expire and Stored bytes: 0.

2. Create a log stream

   Normally the agent creates streams for you. Here you’re sending lines by hand, so you make one.

   AWSIn the AWS console: CloudWatch→Log groups→/quickcart/orders→Log streams→Create log stream

   Field	Enter
   Log stream name	lab-1

   Choose Create.

   You should see

   lab-1 in the list of log streams, with no events yet.

3. Create the metric filter before any logs arrive

   A metric filter only counts lines that arrive after it exists, so create it first.

   AWSIn the AWS console: CloudWatch→Log groups→/quickcart/orders→Metric filters→Create metric filter

   Field	Enter
   Filter pattern	{ $.level = "ERROR" }

   Choose Next.

   Field	Enter
   Filter name	quickcart-app-errors
   Metric namespace	QuickCart/App
   Metric name	AppErrorCount
   Metric value	1
   Default value	0
   Unit	leave empty

   Choose Next, then Create metric filter.

   You should see

   The filter listed on the Metric filters tab, showing the pattern and QuickCart/App / AppErrorCount.

4. Write the log lines to a file

   The console has no way to send log lines by hand, so this step and the next use the CLI.

   First, get the current time. CloudWatch refuses log events more than 14 days old or more than 2 hours in the future, so the timestamps must be close to now.

   TerminalCopy

   date +%s

   Part	What it means
   date	print the date and time
   +%s	in this format: seconds since 1 January 1970, which is how computers count time

   You should see

   1790349400

   Yours will be a different, larger number. CloudWatch wants milliseconds, so add three zeros to the end of yours: 1790349400 becomes 1790349400000.

   In a new folder, create events.json. Replace every highlighted 1790349400000 with your own number. Most editors can do this with find and replace.

   events.jsonCopy

   [
     {"timestamp": 1790349400000, "message": "{\"level\":\"INFO\",\"route\":\"/orders\",\"status\":200,\"latency_ms\":41,\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"INFO\",\"route\":\"/orders\",\"status\":200,\"latency_ms\":38,\"instance_id\":\"i-0bb22\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":6,\"error_type\":\"ProgrammingError\",\"instance_id\":\"i-0aa11\",\"detail\":\"Table 'quickcart.orders' doesn't exist\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":5,\"error_type\":\"ProgrammingError\",\"instance_id\":\"i-0bb22\",\"detail\":\"Table 'quickcart.orders' doesn't exist\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"INFO\",\"route\":\"/orders\",\"status\":200,\"latency_ms\":44,\"instance_id\":\"i-0aa11\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3004,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0aa11\",\"detail\":\"Can't connect to MySQL server on 'quickcart-db' (timed out)\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3003,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0bb22\",\"detail\":\"Can't connect to MySQL server on 'quickcart-db' (timed out)\"}"},
     {"timestamp": 1790349400000, "message": "{\"level\":\"ERROR\",\"route\":\"/orders\",\"status\":500,\"latency_ms\":3005,\"error_type\":\"OperationalError\",\"instance_id\":\"i-0aa11\",\"detail\":\"Can't connect to MySQL server on 'quickcart-db' (timed out)\"}"}
   ]

   Part	What it means
   [ and ]	a list of log events
   timestamp	when the event happened, in milliseconds
   message	the log line itself. Here each line is JSON, written inside a string, so its own quotes are escaped as \"

   Eight events: three normal requests, two failures from a missing table, and three timeouts to the database. They’re the kind of lines QuickCart’s app writes.

5. Send them to CloudWatch

   From the folder that holds events.json:

   TerminalCopy

   aws logs put-log-events --region us-east-1 --log-group-name /quickcart/orders --log-stream-name lab-1 --log-events file://events.json

   Part	What it means
   aws logs put-log-events	send log events to CloudWatch Logs
   --region us-east-1	the region the log group is in
   --log-group-name /quickcart/orders	which log group
   --log-stream-name lab-1	which stream in that group
   --log-events file://events.json	read the events from this file. file:// tells the CLI it’s a file, not the events themselves

   You should see

   A short JSON reply. It may include a nextSequenceToken, which you can ignore — CloudWatch no longer needs it. If the reply includes rejectedLogEventsInfo, some events were refused: see the mistake below.

   Open CloudWatch → Log groups → /quickcart/orders → lab-1 to see the eight lines. Choose one to expand it: CloudWatch shows the JSON fields laid out.

6. Ask the logs questions

   AWSIn the AWS console: CloudWatch→Logs Insights

   If the page offers a choice of query language, choose Logs Insights QL. In Select log groups, choose /quickcart/orders. Set the time range to 3h, so it includes your timestamps.

   Paste this query and choose Run query:

   Logs Insights queryCopy

   fields @timestamp, level, error_type, latency_ms, instance_id
   | sort @timestamp desc
   | limit 20

   Part	What it means
   fields @timestamp, level, error_type, latency_ms, instance_id	show these fields as columns. All but @timestamp come from your JSON
   | sort @timestamp desc	newest first
   | limit 20	at most 20 rows

   You should see

   Eight rows with the fields in their own columns. error_type is empty for the three normal requests.

   Now count the errors by type:

   Logs Insights queryCopy

   filter level = "ERROR"
   | stats count() as errors, avg(latency_ms) as avg_ms by error_type

   Part	What it means
   filter level = "ERROR"	keep only the error lines
   | stats count() as errors	count them, in a column called errors
   avg(latency_ms) as avg_ms	and average their latency_ms, in a column called avg_ms
   by error_type	one row for each different error_type

   Logs Insights shows

   error_type          errors   avg_ms
   OperationalError    3        3004
   ProgrammingError    2        5.5

   The rows may come in the other order. This is the question plain-text logs couldn’t answer. Two different problems: one fails instantly with a clear message, the other waits 3 seconds for a database that never answers — the sign of a network block, not a code bug.

7. See the metric the filter made

   AWSIn the AWS console: CloudWatch→Metrics→All metrics→Browse→QuickCart/App→Metrics with no dimensions

   Tick AppErrorCount. On the Graphed metrics tab, set Statistic to Sum and Period to 1 minute.

   You should see

   One point at 5 — the five error lines — at the minute of your timestamp. The namespace can take a few minutes to appear the first time. This metric is what the next lesson puts an alarm on.

8. Set retention

   AWSIn the AWS console: CloudWatch→Log groups→/quickcart/orders→Actions→Edit retention setting(s)

   Field	Enter
   Expire events after	1 week

   Choose Save.

   You should see

   Retention: 1 week in the log group list. For a real app, 30 days is a common choice for application logs; audit logs are often kept much longer, by law.

9. Clean-up step: Clean up

   AWSIn the AWS console: CloudWatch→Log groups→/quickcart/orders→Actions→Delete log group(s)

   Choose Delete. This deletes the stream, its events and the metric filter with it.

   You should see

   The log group is gone from the list. The AppErrorCount metric may stay visible for a while, but it receives no new data. You can’t delete a metric by hand; CloudWatch removes it after it has had no data for 15 months, and it costs nothing while nothing is sent to it.

   If you’re going straight on to Alarms that actually fire, you can keep the log group instead — that lab starts by building the same thing.