Continuous integration

Continuous integration, or CI, means every change is checked automatically, the same way, before it joins main. Not on someone’s laptop, but on a clean agent, from the exact commit that will be merged.

What runs, and in what order

QuickCart’s pipeline has five gates. They are ordered cheapest first, so the most common mistakes are reported in seconds.

Fail fast means three things. Put quick checks first. Stop at the first failure; declarative pipelines do this for you. And when someone pushes again to the same pull request, abort the build that’s still running for the old commit, because its answer no longer matters. disableConcurrentBuilds(abortPrevious: true) does that last one.

One job per branch: multibranch pipelines

A multibranch pipeline is a Jenkins job that looks at a repository and makes a job for every branch and every pull request that has a Jenkinsfile. Each one builds with the Jenkinsfile from its own branch. New branch, new job; branch deleted, job removed.

The GitHub Branch Source plugin, part of the suggested plugins, does this for GitHub. It also reports each build’s result back to GitHub as a status check on the commit. For a pull request, the check is called continuous-integration/jenkins/pr-merge.

How Jenkins hears about a push: webhooks

Jenkins can ask GitHub every few minutes whether anything changed. That’s called polling, and it’s slow and wasteful. A webhook is the other way round: the moment someone pushes, GitHub sends a message to Jenkins.

For QuickCart’s Jenkins on EC2, that means GitHub must be able to reach Jenkins. So the controller gets a load balancer with HTTPS in front of it, whose security group only accepts connections from GitHub’s webhook address ranges, which GitHub publishes. Then, in the repository on GitHub:

Settings → Webhooks → Add webhook

The /github-webhook/ path is where the GitHub plugin in Jenkins listens. The secret lets Jenkins check that each message really came from GitHub.

In this lesson’s lab, Jenkins runs on your computer, which GitHub can’t reach. So you’ll use a periodic scan instead: Jenkins checks GitHub every minute. It works the same, just a minute slower.

The gate must be required

A red check does nothing by itself. In GitHub, a ruleset on main makes it binding:

Terraform plan on every pull request

For infrastructure changes, the most useful thing CI can do is run terraform plan and post the result on the pull request. Reviewers then read what will actually change in AWS, not what the author thinks will change. In a real setup, the plan stage uses a read-only AWS role and the shared S3 state from Terraform state. Nothing is applied until after the merge.

Caching: faster without being wrong

Cache things that are downloaded, never things that are checked. A cached test result is a test that didn’t run.

Shared libraries: write it once

With ten services, ten Jenkinsfiles grow the same lint stage, each slightly different. A shared library is a git repository of pipeline code that every Jenkinsfile can load. Every Groovy file in its vars/ folder becomes a step with the file’s name.

def call(Map config = [:]) {
    String dockerfile = config.get('dockerfile', 'Dockerfile')
    String terraformDir = config.get('terraformDir', 'infra')
    sh "hadolint '${dockerfile}'"
    sh "terraform fmt -check -recursive '${terraformDir}'"
}

Register the library once, in Manage Jenkins → System → Global Trusted Pipeline Libraries: a name, quickcart-pipeline-lib, a default version, and the repository. Then any Jenkinsfile can use it:

@Library('quickcart-pipeline-lib@v1') _
pipeline {
    agent any
    stages {
        stage('Lint') {
            steps {
                quickcartLint(terraformDir: 'terraform')
            }
        }
    }
}

Pin a version, like the tag v1, so a change to the library can’t break every service’s build at once. Libraries registered here are trusted: their code runs with full access to Jenkins, so changes to the library repository need the same review as changes to Jenkins itself.

Flaky tests

A flaky test sometimes passes and sometimes fails, with no change to the code. Usually it depends on timing, on the order tests run in, or on something outside the test such as the network. Google has written publicly about how much time flaky tests cost it, even with its tooling. Flaky tests are dangerous because they teach people that red doesn’t mean anything.

Keep retry for things that genuinely fail at random and are safe to repeat, like downloading a file. Never for a test.

Below is the pipeline a QuickCart pull request goes through. Push a commit that has no problems. Then choose A failing test and push again, and try the others.

What to notice: a commit stops at the first gate it fails, and nothing after that gate runs. The cheaper the gate, the sooner you know. A Dockerfile mistake is reported 20 seconds after the push; a vulnerability, only after the image is built.

Free — Jenkins and every tool run on your computer, and a public GitHub repository is free. No AWS.

You’ll put QuickCart’s orders service in a GitHub repository, point a multibranch pipeline at it, open a pull request that gets a Terraform plan posted on it, then break a test and watch the merge get blocked. About 60 minutes.

You need the Jenkins on your computer from The Jenkinsfile, a GitHub account, and Docker running.

1. Install the tools the pipeline uses

   The pipeline runs on your computer, so the tools it calls must be installed there:

   TerminalCopy

   brew install node hadolint trivy gh

   Part	What it means
   brew install	install with Homebrew
   node	runs the orders service and its tests
   hadolint	checks Dockerfiles for common mistakes
   trivy	scans images for known vulnerabilities
   Terraform	for the plan. You installed it from HashiCorp’s tap in Set up your computer; Homebrew’s plain terraform is an old version
   gh	GitHub’s command-line tool, used to post the plan on the pull request

   Jenkins only finds tools that are on the PATH of the terminal it was started from. So stop Jenkins with Ctrl+C in its terminal, and start it again in a new terminal window:

   TerminalCopy

   jenkins-lts --httpListenAddress=127.0.0.1 --httpPort=8080

   Part	What it means
   jenkins-lts	start Jenkins, which now inherits a PATH that includes the new tools
   --httpListenAddress=127.0.0.1	only this computer can reach it
   --httpPort=8080	the port for the web page

2. Create the repository and a token

   On GitHub, choose New repository:

   Field	Enter
   Repository name	quickcart-orders
   Visibility	Public. Required-check rules are free on public repositories
   Add a README file	ticked

   Choose Create repository. Copy its HTTPS address from the Code button; the next steps call it REPO_URL.

   Jenkins needs a token to read the repository, post comments and report checks. On GitHub, go to your profile picture → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token:

   Field	Enter
   Token name	jenkins-quickcart
   Expiration	30 days
   Repository access	Only select repositories: quickcart-orders
   Permissions → Contents	Read-only
   Permissions → Pull requests	Read and write
   Permissions → Commit statuses	Read and write

   Choose Generate token and copy it. GitHub shows it only once.

3. Add the code

   Clone the repository into your home folder and move into it:

   TerminalCopy

   git clone REPO_URL

   Part	What it means
   git clone	copy a repository from GitHub to your computer
   REPO_URL	the HTTPS address you copied

   TerminalCopy

   cd quickcart-orders

   Part	What it means
   cd quickcart-orders	move into the copy

   Create these files, in these folders. First the code, a tiny function that adds up an order:

   src/total.mjsCopy

   export function orderTotal(items, deliveryFeeCents) {
     const subtotal = items.reduce((sum, item) => sum + item.priceCents * item.quantity, 0);
     return subtotal + deliveryFeeCents;
   }

   Its tests, using the test runner built into Node:

   test/total.test.mjsCopy

   import { test } from "node:test";
   import assert from "node:assert/strict";
   import { orderTotal } from "../src/total.mjs";
   
   const items = [
     { priceCents: 450, quantity: 2 },
     { priceCents: 100, quantity: 1 },
   ];
   
   test("orderTotal adds up the items", () => {
     assert.equal(orderTotal(items, 0), 1000);
   });
   
   test("orderTotal adds the delivery fee", () => {
     assert.equal(orderTotal(items, 250), 1250);
   });

   A small web server around it, which the next lesson deploys:

   server.mjsCopy

   import { createServer } from "node:http";
   import { orderTotal } from "./src/total.mjs";
   
   const version = process.env.APP_VERSION || "dev";
   
   const server = createServer((req, res) => {
     res.setHeader("Content-Type", "application/json");
     if (req.url === "/health") {
       res.end(JSON.stringify({ status: "ok", version }));
       return;
     }
     const example = orderTotal([{ priceCents: 450, quantity: 2 }], 250);
     res.end(JSON.stringify({ service: "orders", version, exampleTotalCents: example }));
   });
   
   server.listen(8080, () => {
     console.log(`orders ${version} listening on port 8080`);
   });

   Its Dockerfile:

   DockerfileCopy

   FROM node:24-alpine
   WORKDIR /app
   COPY src ./src
   COPY server.mjs ./
   ENV NODE_ENV=production
   USER 1000
   EXPOSE 8080
   CMD ["node", "server.mjs"]

   USER 1000 runs the server as the image’s built-in node user, by number rather than name, so any system can resolve it. Writing a Dockerfile explains every other line.

   And a small piece of Terraform, in a folder called infra. It uses terraform_data, a resource built into Terraform that stores values and creates nothing in AWS, so the plan runs without an AWS login:

   main.tfCopy

   terraform {
     required_version = ">= 1.10.0"
   }
   
   variable "environment" {
     description = "Which environment this plan is for"
     type        = string
     default     = "dev"
   }
   
   resource "terraform_data" "orders_service" {
     input = {
       environment   = var.environment
       desired_count = 2
       cpu           = 256
     }
   }

   Setting	What it does	If you leave it out	When you’d change it
   required_version	Refuse Terraform older than 1.10	Any version is accepted	When you rely on newer features
   variable "environment" → description	A note for whoever sets it	No note	—
   type = string	Only text is accepted	Any type is accepted	—
   default = "dev"	The value if nobody sets one	Terraform asks for it, and the pipeline has nobody to answer	—
   terraform_data → input	Values stored in state, standing in for a real ECS service’s settings	Nothing is stored	This is what the pull request will change

   Before you push, run the same checks the pipeline will run:

   TerminalCopy

   node --test

   Part	What it means
   node --test	find files named like *.test.mjs and run the tests in them

   You should see

   ✔ orderTotal adds up the items (0.79ms)
   ✔ orderTotal adds the delivery fee (0.14ms)
   ℹ tests 2
   ℹ suites 0
   ℹ pass 2
   ℹ fail 0

   Your times will differ.

4. Add the Jenkinsfile and push

   In the root of the repository:

   JenkinsfileCopy

   pipeline {
       agent any
   
       options {
           timeout(time: 20, unit: 'MINUTES')
           timestamps()
           disableConcurrentBuilds(abortPrevious: true)
           buildDiscarder(logRotator(numToKeepStr: '30'))
       }
   
       environment {
           TF_PLUGIN_CACHE_DIR = "${env.HOME}/.terraform.d/plugin-cache"
       }
   
       stages {
           stage('Lint') {
               steps {
                   sh 'hadolint Dockerfile'
                   sh 'terraform fmt -check -recursive infra'
               }
           }
   
           stage('Unit tests') {
               steps {
                   sh 'node --test --test-reporter=spec --test-reporter-destination=stdout --test-reporter=junit --test-reporter-destination=test-results.xml'
               }
               post {
                   always {
                       junit 'test-results.xml'
                   }
               }
           }
   
           stage('Build image') {
               steps {
                   sh 'docker build -t "quickcart-orders:$GIT_COMMIT" .'
               }
           }
   
           stage('Scan image') {
               steps {
                   sh 'trivy image --exit-code 1 --severity CRITICAL --ignore-unfixed --no-progress "quickcart-orders:$GIT_COMMIT"'
               }
           }
   
           stage('Terraform plan') {
               when {
                   changeRequest()
               }
               steps {
                   dir('infra') {
                       sh 'mkdir -p "$TF_PLUGIN_CACHE_DIR"'
                       sh 'terraform init -input=false -no-color'
                       sh 'terraform plan -input=false -no-color -out=tfplan'
                       sh 'terraform show -no-color tfplan > plan.txt'
                   }
                   withCredentials([usernamePassword(credentialsId: 'github-token', usernameVariable: 'GH_USER', passwordVariable: 'GH_TOKEN')]) {
                       sh 'gh pr comment "$CHANGE_ID" --body-file infra/plan.txt'
                   }
               }
           }
       }
   
       post {
           failure {
               echo "Build failed: ${env.BUILD_URL}"
           }
           cleanup {
               deleteDir()
           }
       }
   }

   Line	What it means
   disableConcurrentBuilds(abortPrevious: true)	A new push to the same branch aborts the build still running for the old commit
   TF_PLUGIN_CACHE_DIR = "${env.HOME}/…"	Keep downloaded Terraform providers in one folder that every build reuses. Groovy fills in your home folder
   hadolint Dockerfile	Lint the Dockerfile. Any finding fails the stage
   terraform fmt -check -recursive infra	Fail if any .tf file in infra isn’t formatted. -check reports without changing files
   node --test --test-reporter=spec --test-reporter-destination=stdout	Run the tests, printing readable results in the log
   --test-reporter=junit --test-reporter-destination=test-results.xml	Also write the results as JUnit XML, the format Jenkins understands
   junit 'test-results.xml' in post → always	Read the results even when tests failed, so the build page lists which ones
   docker build -t "quickcart-orders:$GIT_COMMIT" .	Build the image, tagged with the commit being built. GIT_COMMIT is set by Jenkins’ checkout
   trivy image	Scan the image’s operating system packages and libraries against a list of known vulnerabilities
   --exit-code 1 --severity CRITICAL	Fail the stage only for critical findings. Start strict and narrow; widen it to HIGH once you’re clean
   --ignore-unfixed	Skip vulnerabilities that have no fix yet, since there’s nothing you could upgrade to
   --no-progress	No progress bar, which only clutters a build log
   when { changeRequest() }	The plan runs only for pull requests
   dir('infra') { }	Run the steps inside from the infra folder
   terraform plan … -out=tfplan	Work out the plan and save it to a file
   terraform show -no-color tfplan > plan.txt	Turn the saved plan into readable text. > writes that text into plan.txt instead of the log
   usernamePassword(credentialsId: 'github-token', …)	Fetch the GitHub token you’ll store in Jenkins, into GH_TOKEN, which gh reads
   gh pr comment "$CHANGE_ID" --body-file infra/plan.txt	Comment on the pull request whose number is in CHANGE_ID, with the plan as the text

   Commit and push it all to main:

   TerminalCopy

   git add .

   Part	What it means
   git add .	include every new file in the next commit

   TerminalCopy

   git commit -m "Orders service with a CI pipeline"

   Part	What it means
   git commit -m "…"	record the files, with this message

   TerminalCopy

   git push

   Part	What it means
   git push	send the commit to GitHub

5. Create the multibranch pipeline

   In Jenkins, choose New Item:

   Field	Enter
   Enter an item name	quickcart-orders
   Type	Multibranch Pipeline

   Choose OK. Under Branch Sources, choose Add source → GitHub. Next to Credentials, choose Add → Jenkins:

   Field	Enter
   Kind	Username with password
   Username	your GitHub username
   Password	the token you generated
   ID	github-token
   Description	GitHub token for quickcart-orders

   Choose Add, then pick the new credential in the Credentials list and fill in the rest:

   Field	Enter
   Repository HTTPS URL	your REPO_URL
   Behaviours	Leave Discover branches and Discover pull requests from origin. Delete Discover pull requests from forks
   Build Configuration → Mode	by Jenkinsfile, Script Path Jenkinsfile
   Scan Multibranch Pipeline Triggers	Tick Periodically if not otherwise run, Interval 1 minute

   Choose Save.

   You should see

   The Scan Repository Log finds the main branch and its Jenkinsfile, and schedules a build. After a minute or two, the job shows a main job with build #1. Its stages all pass except Terraform plan, which is skipped, because main isn’t a pull request. The first run is slow while Trivy downloads its vulnerability database.

6. Open a pull request and get a plan

   Make a branch:

   TerminalCopy

   git switch -c more-orders-capacity

   Part	What it means
   git switch -c	create a branch and move to it
   more-orders-capacity	its name

   In infra/main.tf, change desired_count = 2 to desired_count = 3. Commit and push the branch:

   TerminalCopy

   git commit -am "Run three orders tasks"

   Part	What it means
   git commit	record a change
   -a	include every file git already knows that you’ve changed
   -m "…"	the message

   TerminalCopy

   git push -u origin more-orders-capacity

   Part	What it means
   git push	send it to GitHub
   -u origin more-orders-capacity	create the branch on GitHub too, and remember the link

   On GitHub, choose Compare & pull request, then Create pull request.

   You should see

   Within a minute or two, Jenkins shows a Pull Requests tab with PR-1. On GitHub, the pull request gets a comment from your account containing the plan, and a check called continuous-integration/jenkins/pr-merge that turns green.

   The plan comment includes

     # terraform_data.orders_service will be created
     + resource "terraform_data" "orders_service" {
         + id     = (known after apply)
         + input  = {
             + cpu           = 256
             + desired_count = 3
             + environment   = "dev"
           }
         + output = (known after apply)
       }
   
   Plan: 1 to add, 0 to change, 0 to destroy.

   It says create because this lab’s Terraform has no shared state, so every plan starts from nothing. With the S3 state from Terraform state, the same change would show ~ update in-place and only the desired_count line.

7. Make the check required

   On GitHub, in the repository, go to Settings → Rules → Rulesets → New ruleset → New branch ruleset:

   Field	Enter
   Ruleset Name	protect-main
   Enforcement status	Active
   Target branches	Add target → Include default branch
   Require a pull request before merging	ticked
   Require status checks to pass	ticked. Add checks → continuous-integration/jenkins/pr-merge

   Choose Create.

8. Break a test

   On the same branch, in src/total.mjs, change the last line of the function to forget the fee:

   src/total.mjsCopy

   export function orderTotal(items, deliveryFeeCents) {
     const subtotal = items.reduce((sum, item) => sum + item.priceCents * item.quantity, 0);
     return subtotal;
   }

   TerminalCopy

   git commit -am "Simplify the total"

   Part	What it means
   git commit -am "…"	record the change to a file git already knows, with this message

   TerminalCopy

   git push

   Part	What it means
   git push	send it. The branch already knows where to go

   Build #2 of PR-1 includes

   ✖ orderTotal adds the delivery fee
     AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
   
     1000 !== 1250
   
   Stage "Build image" skipped due to earlier failure(s)
   Stage "Scan image" skipped due to earlier failure(s)
   Stage "Terraform plan" skipped due to earlier failure(s)
   Finished: FAILURE

   You should see

   On GitHub, the check is red and the merge button says merging is blocked. On the build page in Jenkins, Test Result lists the one failing test by name, because junit read the report.

   Put the + deliveryFeeCents back, commit and push. The check goes green and the pull request can be merged. Merge it.

9. Clean-up step: Clean up

   Nothing here costs money. Keep the repository, the token and the quickcart-orders job: Continuous delivery builds on them.

   When you’ve finished with them, delete the job in Jenkins (quickcart-orders → Delete Multibranch Pipeline), delete the token on GitHub (Settings → Developer settings → Fine-grained tokens → Delete), and free the disk space the test images use:

   TerminalCopy

   docker image prune -a

   Part	What it means
   docker image prune	delete images no container is using
   -a	all unused images, not only untagged ones. Docker asks you to confirm