Canary from a pipeline

A canary sends a small share of real traffic to a new version, watches it, and either widens the share step by step or pulls it back. Ways to release explained the idea. This lesson puts it in the pipeline, so a machine makes the call using rules people agreed on calmly, in advance.

Abort criteria, written first

Before the release, the team writes down what “failing” means. Each criterion is a number the pipeline can check.

How the traffic moves

Newer ECS releases also have canary options built into ECS deployments. Check what your platform supports before building your own.

Where the numbers come from

The pipeline reads the same metrics a person would, but per version, so the canary is judged only on its own requests.

The pipeline’s shape

Putting the rollback in post { failure } means it happens on any failure, not just the ones you thought of.

Below, a pipeline releases a new version of QuickCart’s menu service in steps. First release a Healthy version. Then choose Raises errors and start again. Finally try Quiet night, where very little traffic arrives.

Waiting to start.

What to notice: nobody decides anything during the canary. The failing version is stopped at the first step, and only that step’s share of customers ever saw it. On a quiet night, the pipeline refuses to call a handful of successful requests a pass; it waits until it has enough to judge.

Pennies — a few hundred calls to a tiny Lambda function: well under a cent, and inside the free allowance on most accounts.

You’ll give Jenkins a canary to run. The service is the quickcart-menu Lambda function from Ways to release, with a live alias that can send a share of calls to a second version. The pipeline shifts 25% and then 50% to the new version. At each step it calls the alias until enough calls reach the new version, then compares that version’s error rate with the limit. About 40 minutes.

You need the Jenkins on your computer from The Jenkinsfile, started with AWS_PROFILE=quickcart as in Continuous delivery.

1. Create the function and version 1

   If you still have quickcart-menu from Ways to release, delete it first (Lambda → Functions → quickcart-menu → Actions → Delete function), so the version numbers below match.

   AWSIn the AWS console: Lambda→Functions→Create function

   Field	Enter
   Create function	Author from scratch
   Function name	quickcart-menu
   Runtime	The newest Node.js version offered
   Architecture	arm64

   Choose Create function. In the Code tab, replace the contents of index.mjs with version 1, then choose Deploy:

   index.mjsCopy

   export const handler = async () => {
     return { version: 1, dishes: ["Margherita", "Pad thai", "Falafel wrap"] };
   };

   Publish it as a version:

   AWSIn the AWS console: Lambda→Functions→quickcart-menu→Actions→Publish new version

   Field	Enter
   Version description	stable

   Then create the alias that customers call:

   AWSIn the AWS console: Lambda→Functions→quickcart-menu→Aliases→Create alias

   Field	Enter
   Name	live
   Version	1

   You should see

   The alias live pointing at version 1, with no weighted version.

2. Publish a version that fails

   Replace index.mjs with version 2. It fails on about 4 calls in 10. Choose Deploy:

   index.mjsCopy

   export const handler = async () => {
     if (Math.random() < 0.4) {
       throw new Error("menu lookup failed");
     }
     return { version: 2, dishes: ["Margherita", "Pad thai", "Falafel wrap", "Ramen"] };
   };

   Publish it: Actions → Publish new version, description adds ramen, then Publish.

   You should see

   Versions lists 1 and 2. The live alias still sends every call to version 1: publishing a version changes nothing for customers.

3. Create the canary pipeline

   In Jenkins, choose New Item:

   Field	Enter
   Enter an item name	quickcart-canary
   Type	Pipeline

   Choose OK, and paste this into Pipeline → Script:

   JenkinsfileCopy

   pipeline {
       agent any
   
       parameters {
           string(name: 'STABLE', defaultValue: '1', description: 'The version customers get today')
           string(name: 'CANARY', defaultValue: '2', description: 'The new version to try')
       }
   
       options {
           timeout(time: 20, unit: 'MINUTES')
           disableConcurrentBuilds()
           timestamps()
       }
   
       environment {
           AWS_REGION = 'us-east-1'
           AWS_PAGER  = ''
           FUNCTION   = 'quickcart-menu'
           ALIAS      = 'live'
       }
   
       stages {
           stage('Check the versions') {
               steps {
                   script {
                       if (params.STABLE == params.CANARY) {
                           error('STABLE and CANARY must be different versions')
                       }
                   }
                   sh 'aws lambda get-function --function-name "$FUNCTION" --qualifier "$CANARY" --query Configuration.Version --output text'
               }
           }
           stage('Canary at 25%') {
               steps {
                   script {
                       shift(0.25)
                       judge()
                   }
               }
           }
           stage('Canary at 50%') {
               steps {
                   script {
                       shift(0.5)
                       judge()
                   }
               }
           }
           stage('Promote') {
               steps {
                   sh 'aws lambda update-alias --function-name "$FUNCTION" --name "$ALIAS" --function-version "$CANARY" --routing-config \'{"AdditionalVersionWeights":{}}\' --query FunctionVersion --output text'
                   echo "Version ${params.CANARY} now takes all traffic. Version ${params.STABLE} is still there for a quick rollback."
               }
           }
       }
   
       post {
           failure {
               echo "Rolling back: all traffic to version ${params.STABLE}"
               sh 'aws lambda update-alias --function-name "$FUNCTION" --name "$ALIAS" --function-version "$STABLE" --routing-config \'{"AdditionalVersionWeights":{}}\' --query FunctionVersion --output text'
           }
       }
   }
   
   def shift(double share) {
       def weights = '{"AdditionalVersionWeights":{"' + params.CANARY + '":' + share + '}}'
       echo "Sending ${share * 100}% of calls to version ${params.CANARY}"
       sh "aws lambda update-alias --function-name \"\$FUNCTION\" --name \"\$ALIAS\" --function-version \"\$STABLE\" --routing-config '${weights}' --query RoutingConfig --output json"
   }
   
   def judge() {
       int minCalls = 10
       int maxTries = 150
       double maxErrorPercent = 2.0
       int canaryCalls = 0
       int canaryErrors = 0
       int tries = 0
       while (canaryCalls < minCalls && tries < maxTries) {
           def result = sh(
               script: 'aws lambda invoke --function-name "$FUNCTION" --qualifier "$ALIAS" --query "[ExecutedVersion, FunctionError]" --output text /dev/null',
               returnStdout: true
           ).trim().split(/\s+/)
           tries++
           if (result[0] == params.CANARY) {
               canaryCalls++
               if (result.length > 1 && result[1] != 'None') {
                   canaryErrors++
               }
           }
       }
       if (canaryCalls < minCalls) {
           error("Only ${canaryCalls} of ${tries} calls reached version ${params.CANARY}. Too few to judge, so not promoting.")
       }
       double errorPercent = canaryErrors * 100.0 / canaryCalls
       echo "Version ${params.CANARY}: ${canaryCalls} calls, ${canaryErrors} errors (${Math.round(errorPercent)}%). Limit: ${maxErrorPercent}%."
       if (errorPercent > maxErrorPercent) {
           error("Abort: version ${params.CANARY} failed ${Math.round(errorPercent)}% of its calls, above the ${maxErrorPercent}% limit")
       }
   }

   Line	What it does	If you leave it out
   parameters → STABLE, CANARY	Which version customers get now, and which one to try. Jenkins also sets them as environment variables	You’d edit the pipeline for every release
   timeout(time: 20, …)	Ends the build, and so triggers the rollback, if it runs too long	A hung call would leave the canary running at 25% indefinitely
   disableConcurrentBuilds()	One canary at a time	Two builds would fight over the same alias
   Check the versions	Stops at once if both versions are the same, or the canary version doesn’t exist	The canary step would fail halfway, after moving traffic
   get-function … --qualifier "$CANARY"	Fails the build if that version doesn’t exist	—
   shift(0.25)	Sets the alias to version STABLE, sending 25% of calls to CANARY	—
   --routing-config '{"AdditionalVersionWeights":{"2":0.25}}'	The weighted part: which extra version gets calls, and what share, from 0 to 1	All calls stay on the stable version
   judge()	Calls the alias, counts only the calls that reached the new version, and fails the build if too few arrived or too many failed	Traffic would widen with nobody checking
   aws lambda invoke … --qualifier "$ALIAS"	Calls the function through live, exactly as a customer would	Calling a version directly would bypass the weights
   --query "[ExecutedVersion, FunctionError]" --output text	Prints which version answered, and Unhandled if it threw an error, or None if not	You couldn’t tell the versions apart
   /dev/null	Where invoke writes the function’s reply. The pipeline doesn’t need it	Required: invoke always takes an output file
   minCalls = 10, maxTries = 150	Needs 10 calls to the new version, and gives up after 150 tries	Without a minimum, four lucky calls could pass a broken version
   maxErrorPercent = 2.0	The abort criterion, written down before the release	—
   Promote	Points live at the new version with no weights: 100% of calls	The canary would stay at 50%
   post → failure	Points live back at STABLE with no weights, whatever made the build fail	A failed canary would leave customers on the bad version

   Choose Save.

4. Release the failing version

   Choose Build Now (Jenkins uses the defaults, 1 and 2, the first time). Open Console Output. Within a minute or two:

   The console output, shortened

   Sending 25.0% of calls to version 2
   Version 2: 10 calls, 4 errors (40%). Limit: 2.0%.
   ERROR: Abort: version 2 failed 40% of its calls, above the 2.0% limit
   Rolling back: all traffic to version 1
   1
   Finished: FAILURE

   Your error count will differ, because version 2 fails at random. It is very unlikely to stay under the limit.

   You should see

   In the console, Lambda → quickcart-menu → Aliases → live points at version 1 with no weighted version. The stage view shows Canary at 25% failed and Canary at 50% and Promote never ran. Nobody decided anything; the rule did.

5. Fix it, and release again

   Replace index.mjs with a fixed version, choose Deploy, and publish it as version 3 with the description adds ramen, fixed:

   index.mjsCopy

   export const handler = async () => {
     return { version: 3, dishes: ["Margherita", "Pad thai", "Falafel wrap", "Ramen"] };
   };

   In Jenkins, choose Build with Parameters, keep STABLE 1, and set CANARY to 3.

   The console output, shortened

   Sending 25.0% of calls to version 3
   Version 3: 10 calls, 0 errors (0%). Limit: 2.0%.
   Sending 50.0% of calls to version 3
   Version 3: 10 calls, 0 errors (0%). Limit: 2.0%.
   3
   Version 3 now takes all traffic. Version 1 is still there for a quick rollback.
   Finished: SUCCESS

   You should see

   The alias live points at version 3 with no weights. Every customer now gets the fixed version, and it got there without anyone watching a graph.

6. Clean-up step: Clean up

   AWSIn the AWS console: Lambda→Functions→quickcart-menu→Actions→Delete function

   Type confirm if asked, and choose Delete. Deleting the function deletes its versions and the live alias too.

   Lambda also made a log group, which keeps logs until you delete it:

   AWSIn the AWS console: CloudWatch→Log groups→/aws/lambda/quickcart-menu→Actions→Delete log group(s)

   The IAM role Lambda created, named like quickcart-menu-role- followed by random letters, costs nothing. You can delete it in IAM → Roles to keep things tidy.

   In Jenkins, open quickcart-canary and choose Delete Pipeline. If you’ve finished the Automation module, stop Jenkins with Ctrl+C in its terminal.

   You should see

   No quickcart-menu function or log group remains. Nothing from this lab is left costing money.