Releases without downtime

QuickCart wants to release in the middle of a busy lunchtime with zero failed requests, and to undo a broken release in minutes without anyone having to be awake. This lesson finds the three moments where requests get dropped, closes each one, and proves it with a count.

A rolling deployment replaces the old copies of your app with new ones, a few at a time, while the service keeps running. On ECS, pointing a service at a new task definition revision starts one.

It looks seamless from a distance. Close up, every task goes through a life: it starts, becomes ready, serves traffic, is removed from the load balancer, and is stopped. Requests fail at the edges of that life.

The three moments, and the three fixes

Rolling settings: how many at once

Two numbers control how ECS swaps tasks, both as a percentage of the desired count. The console calls them Min running tasks % and Max running tasks %; in the ECS API they’re minimumHealthyPercent and maximumPercent, part of the service’s deployment configuration.

100 and 200 is the safe choice. New tasks start first, and old ones are only removed once the new ones are healthy. You briefly pay for four tasks instead of two.

Moment 1: starting, and being ready

The load balancer decides whether a target gets traffic from its health check. So the health check must answer one question honestly: can this copy serve an order right now? QuickCart’s /health only answers once gunicorn and the app have loaded. A check that only tests whether a port is open can pass while the app is still loading.

A separate setting protects slow starters. The health check grace period tells ECS to ignore failed load balancer health checks for the first few seconds of a task’s life. Without it, a task that takes 40 seconds to warm up is marked unhealthy and replaced, over and over, and the release never finishes.

Moment 2: leaving the load balancer

When ECS removes a task, it first deregisters it from the target group. The load balancer stops sending it new requests, but lets the ones already in progress carry on for up to the deregistration delay. When they’ve finished, or the delay runs out, ECS stops the task.

The default delay is 300 seconds. That’s safe, but it makes every release wait five minutes per batch of tasks. QuickCart’s slowest request takes a few seconds, so 30 seconds is plenty. Choose a number longer than your slowest normal request.

Moment 3: stopping

To stop a task, ECS sends its container SIGTERM, the polite “please stop” signal. It then waits for the stop timeout, 30 seconds by default on Fargate and at most 120, and if the process is still running it sends SIGKILL, which can’t be refused.

A well-behaved app hears SIGTERM, stops accepting new work, finishes what it’s doing and exits. gunicorn does this: --graceful-timeout 20 gives its workers up to 20 seconds to finish. That has to be shorter than the stop timeout, or gunicorn is killed before it’s done.

The square brackets are the exec form. The Dockerfile lesson covers it in full.

When a release is broken: the circuit breaker

If the new version can’t start, or keeps failing its health checks, ECS would keep trying for ever. Meanwhile the old tasks are still serving, because the new ones never became healthy, so customers see nothing. But the release is stuck, and someone has to notice.

The deployment circuit breaker notices for you. After a number of failed tasks, which depends on the desired count and is at least a few, it marks the release as failed. With rollback turned on, it returns the service to the last deployment that completed successfully.

Changing the database during a release

During a rolling release, old and new versions run at the same time, against the same database. So a database change must not break the old version. Do it in three separate releases:

Renaming or deleting a column in one step breaks the version that’s still running, instantly.

Below, QuickCart releases v2 while customers send about ten requests a second. First release with all three fixes on. Then switch one fix off at a time, release again, and watch the failed count.

What to notice: each fix protects one moment, and the failures it prevents are different. Readiness protects the start, the drain protects the handover, and graceful shutdown protects the stop. And the results don’t simply add up: with the drain on, a missing graceful shutdown got away with it. On Kubernetes, where removal and stopping happen at the same moment, it wouldn’t.

Costs money — about $2 a day while it runs, the same as ECS on Fargate: two small tasks, the load balancer and their public IP addresses. During each release you briefly pay for four tasks instead of two. Prices are for us-east-1 and change. Delete everything at the end.

You’ll release v2 of the order service while a script sends it traffic, and count the failed requests. Then you’ll switch one fix off and see the difference, and release a broken version to watch the circuit breaker undo it. About 45 minutes.

This lab uses the service from ECS on Fargate: the quickcart cluster, the order-service service with 2 tasks, quickcart-alb and quickcart-orders-tg. If you deleted it, rebuild it with that lesson first. You also need Docker and your order-service folder from Storing images: ECR, and Python 3, which macOS and Ubuntu (including Ubuntu on WSL) already have.

1. Check the service is healthy

   AWSIn the AWS console: Amazon Elastic Container Service→Clusters→quickcart→Services→order-service

   You should see

   2/2 tasks running, and one deployment, marked Completed. In EC2 → Target groups → quickcart-orders-tg → Targets, two healthy targets.

   Copy the load balancer’s DNS name from EC2 → Load balancers → quickcart-alb. It looks like quickcart-alb-1234567890.us-east-1.elb.amazonaws.com. You’ll use it several times.

2. Check the three fixes are in place

   Check each setting where it lives. The ECS lab set most of them already.

   Fix	Where to look	Should be
   Readiness	EC2 → Target groups → quickcart-orders-tg → Health checks	Path /health, healthy threshold 2, interval 10 seconds
   Grace period	ECS → order-service → Configuration and networking (or the service’s Update form)	Health check grace period 30 seconds
   Drain	EC2 → Target groups → quickcart-orders-tg → Attributes	Deregistration delay 30 seconds
   Graceful shutdown	Your Dockerfile	The last line is CMD ["gunicorn", followed by the rest, with square brackets

   Then check the rolling settings and turn on the circuit breaker:

   AWSIn the AWS console: Amazon Elastic Container Service→Clusters→quickcart→Services→order-service→Update service

   Open Deployment options. The console moves sections around from time to time, so find each field by its name:

   Field	Enter
   Deployment type	Rolling update
   Min running tasks %	100
   Max running tasks %	200
   Deployment failure detection	tick Use the Amazon ECS deployment circuit breaker and Rollback on failures

   Choose Update. Some consoles tick the circuit breaker for new services already; if so, leave it on.

   You should see

   A new deployment starts and completes within a few minutes, with the same revision. Changing service settings is itself a small rolling release.

3. Write the load test

   You need a way to count failed requests during a release, not a feeling that “it looked fine”. In your order-service folder, create load.py:

   load.pyCopy

   import sys
   import time
   import urllib.error
   import urllib.request
   
   url = sys.argv[1]
   seconds = int(sys.argv[2]) if len(sys.argv) > 2 else 60
   
   ok = 0
   failed = 0
   end = time.time() + seconds
   
   while time.time() < end:
       try:
           with urllib.request.urlopen(url, timeout=3) as response:
               response.read()
           ok += 1
       except urllib.error.HTTPError as err:
           failed += 1
           print(time.strftime("%H:%M:%S"), "FAILED with HTTP", err.code)
       except (urllib.error.URLError, OSError) as err:
           failed += 1
           reason = getattr(err, "reason", err)
           print(time.strftime("%H:%M:%S"), "FAILED:", reason)
       time.sleep(0.1)
   
   print("----")
   print("requests:", ok + failed, "  ok:", ok, "  failed:", failed)

   Line	What it does
   import lines	load parts of Python’s standard library. Nothing to install
   url = sys.argv[1]	the first thing you type after load.py: the address to test
   seconds = int(sys.argv[2])	the second thing you type: how long to run. If you leave it out, 60 seconds
   while time.time() < end:	keep going until the time is up
   urllib.request.urlopen(url, timeout=3)	fetch the page. Give up after 3 seconds, and count that as a failure
   ok += 1	the page came back with a success code, so add one to ok
   except urllib.error.HTTPError	the load balancer answered with an error code, like 502 or 503. Count it and print the time and code
   except (urllib.error.URLError, OSError)	no answer at all: refused, reset or timed out. Count it and print why
   time.sleep(0.1)	wait a tenth of a second, so it sends roughly ten requests a second
   the last two print lines	the totals, when the time is up

   Try it for 10 seconds. Put your DNS name in place of ALB_DNS_NAME, keeping http:// in front and / after it:

   TerminalCopy

   python3 load.py http://ALB_DNS_NAME/ 10

   Part	What it means
   python3 load.py	run the script with Python 3
   http://ALB_DNS_NAME/	the page to keep asking for: the home page, through the load balancer
   10	run for 10 seconds

   You should see

   ----
   requests: 84   ok: 84   failed: 0

   Your number of requests will differ. It depends on how long each round trip to us-east-1 takes from where you are. What matters is failed: 0.

4. Build and push v2

   The repository is immutable, so v2 needs a new tag. In your order-service folder:

   TerminalCopy

   docker build --platform linux/amd64 --provenance=false --build-arg APP_VERSION=v2 -t order-service:v2 .

   Part	What it means
   docker build	build an image from the Dockerfile in this folder
   --platform linux/amd64	for the x86 processors the task definition asks for
   --provenance=false	a single plain image, so ECR shows one row for it
   --build-arg APP_VERSION=v2	stamp this image as v2. The app prints it, so you can see which version answered
   -t order-service:v2	name it order-service, tag v2
   .	use the files in this folder

   Give it its ECR name, with your account ID:

   TerminalCopy

   docker tag order-service:v2 111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v2

   Part	What it means
   docker tag	give the image a second name. Nothing is copied
   order-service:v2	the image you just built
   the long name after it	the new name: your registry (111122223333.dkr.ecr.us-east-1.amazonaws.com), the repository (quickcart-orders) and the tag (v2)

   TerminalCopy

   docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v2

   Part	What it means
   docker push	upload the image to ECR

   You should see

   Most layers say Layer already exists: only the layers that changed are uploaded. It ends with a line containing v2: digest: sha256:. If it says no basic auth credentials, your ECR login has expired after 12 hours: log in again as in Storing images: ECR.

5. Make revision 2 of the task definition

   AWSIn the AWS console: Amazon Elastic Container Service→Task definitions→order-service→Create new revision→Create new revision with JSON

   Replace everything in the editor with this. It’s revision 1 with two changes: the image tag is now v2, and stopTimeout is written out, so everyone can see how long a task gets to stop. Put your account ID in both highlighted places.

   order-service-task.jsonCopy

   {
     "family": "order-service",
     "requiresCompatibilities": ["FARGATE"],
     "networkMode": "awsvpc",
     "cpu": "256",
     "memory": "512",
     "runtimePlatform": {
       "operatingSystemFamily": "LINUX",
       "cpuArchitecture": "X86_64"
     },
     "executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole",
     "containerDefinitions": [
       {
         "name": "app",
         "image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v2",
         "essential": true,
         "stopTimeout": 30,
         "portMappings": [
           { "containerPort": 8080, "protocol": "tcp" }
         ],
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/order-service",
             "awslogs-region": "us-east-1",
             "awslogs-stream-prefix": "ecs"
           }
         }
       }
     ]
   }

   Setting	What it does	If you leave it out	When you’d change it
   image	Now ends in :v2, the image you just pushed	—	Every release
   stopTimeout	Seconds between SIGTERM and SIGKILL. Must be longer than gunicorn’s --graceful-timeout 20	30 seconds on Fargate, the same as here, but invisible to anyone reading the file	Up to 120 on Fargate, if requests can legitimately take longer

   Every other setting is explained in ECS on Fargate. Choose Create.

   You should see

   order-service:2. Revision 1 is still there, unchanged. That’s what makes going back easy.

6. Release v2 under load

   Open two terminal windows, both in your order-service folder.

   Window 1: start the load test for 4 minutes:

   TerminalCopy

   python3 load.py http://ALB_DNS_NAME/ 240

   Part	What it means
   240	run for 240 seconds: long enough to cover the whole release

   Window 2: watch which version answers, once a second:

   TerminalCopy

   while true; do curl -s ALB_DNS_NAME/; echo; sleep 1; done

   Part	What it means
   while true; do … done	repeat everything between do and done, for ever, until you press Ctrl+C
   curl -s ALB_DNS_NAME/	fetch the home page, which says which version answered. -s hides curl’s progress output
   echo	print a new line, so each answer is on its own line
   sleep 1	wait one second before the next one

   Now start the release, in the console:

   AWSIn the AWS console: Amazon Elastic Container Service→Clusters→quickcart→Services→order-service→Update service

   Field	Enter
   Task definition revision	2 (latest)

   Choose Update, then open the service’s Deployments tab to watch.

   You should see

   Window 2 shows "version":"v1", then a mix of v1 and v2, then only v2. On the Tasks tab, four tasks for a while, then two. After a few minutes the new deployment says Completed.

   When window 1 finishes:

   You should see

   ----
   requests: 1968   ok: 1968   failed: 0

   Your number of requests will differ. Write the numbers down: this is your evidence. Press Ctrl+C in window 2 to stop it.

7. Switch the drain off, and release again

   Now remove one fix to see it matter. Set the deregistration delay to zero:

   AWSIn the AWS console: EC2→Target groups→quickcart-orders-tg→Attributes→Edit

   Field	Enter
   Deregistration delay	0 seconds

   Choose Save changes. Start load.py for 240 seconds again in window 1, then release revision 1 (v1) the same way as the last step. Going back to an older revision is a release like any other.

   You should see

   Some runs show a handful of lines like FAILED with HTTP 502 while the old tasks are stopped, and a failed: total above zero. How many depends on timing and on how busy each task is; a run can occasionally get lucky and show none. With the delay at 30, the same release should show zero.

   Put it back: set the deregistration delay to 30 seconds again.

8. Release a broken version

   Make a revision that can’t possibly start. Open Task definitions → order-service, tick revision 2, and choose Create new revision (the normal form this time). In the container’s Image URI, change the tag at the end to v9-broken. There’s no such image. Choose Create to make revision 3.

   Start load.py for 600 seconds in window 1, then update the service to revision 3. Watch the Deployments and Events tabs.

   You should see

   New tasks start, stop, and are replaced, several times. After a few minutes, the new deployment is marked as failed and a rollback deployment appears, back to the last one that completed. The events mention the circuit breaker and the rollback; the exact wording varies. Through all of it, load.py shows no failures: the old tasks were never removed, because the new ones never became healthy.

   Find out why the tasks failed. On the Tasks tab, change the filter to Stopped and open one of the failed tasks.

   Its stopped reason starts roughly like this

   CannotPullContainerError: pull image manifest has been retried

   The rest of the message says the v9-broken tag was not found. There are no app logs, because the app never started. When a task won’t start, the stopped reason is always the first place to look.

   Finally, remove the broken revision so nobody picks it by accident: tick order-service:3 in Task definitions, then Actions → Deregister.

9. Clean-up step: Clean up

   Everything from ECS on Fargate is still running. Delete it all now, in this order.

   1. The service. ECS → Clusters → quickcart → Services → order-service → Delete service. If the console asks, tick Force delete. Wait until the service and its tasks are gone.

   2. The cluster. ECS → Clusters → quickcart → Delete cluster, then type the confirmation text.

   3. The task definition. Task definitions → order-service: tick every revision, choose Actions → Deregister, then with them still ticked, Actions → Delete.

   4. The load balancer. EC2 → Load balancers: tick quickcart-alb, Actions → Delete load balancer, type confirm.

   5. The target group. EC2 → Target groups: tick quickcart-orders-tg, Actions → Delete.

   6. The security groups. EC2 → Security Groups: delete quickcart-app-sg first, then quickcart-alb-sg. If one won’t delete, wait a minute and try again.

   7. The log group. CloudWatch → Log groups: tick /ecs/order-service, Actions → Delete log group(s).

   8. The image repository, unless you’re going on to Kubernetes and EKS, which uses the same images: Amazon ECR → Repositories, tick quickcart-orders, Delete, and type delete.

   You should see

   ECS → Clusters, EC2 → Load balancers and EC2 → Target groups are all empty. Nothing from this lab is still billing, except the ECR repository if you kept it, which costs pennies a month.