Jenkins from zero
Controllers, agents, credentials and backups.
Part 1: The problem
Theo set up QuickCart’s first Jenkins on a single server. He clicked through the setup, installed a handful of plugins and made the first jobs in the web page. Builds ran on the server itself, two at a time.
It worked for months. Then the team added a job that built container images. Whenever it ran, the Jenkins page took a minute to load, and other builds sat in the queue even though nothing seemed busy.
A few weeks later, someone cleaning up old servers terminated the wrong one. Every job, every build log, and every stored password went with it. Nobody had written down how Jenkins was set up. It took Theo two days to rebuild it from memory, and it was never quite the same.
Both problems come from how Jenkins was set up, not from Jenkins itself: builds ran on the machine that runs everything else, and the only copy of its configuration was on one disk.
This lesson sets Jenkins up the way QuickCart should have from the start: builds on agents, the controller built by Terraform, reachable only through AWS, with its secrets in one place and a backup you’ve tested.
Part 2: The idea
Jenkins is a server that runs your pipelines. Something happens (a push to git, a timer, someone pressing a button) and Jenkins runs the steps you wrote down: test the code, build an image, deploy it. It’s free, open source, and has been around since 2011, so you’ll meet it in a lot of companies.
A Jenkins setup has two kinds of machine.
| Part | What it does | QuickCart has |
|---|---|---|
| Controller | The brain. Serves the web page, decides which build runs where, stores every job, build log and credential | One, on EC2 |
| Agent | The hands. A machine that runs builds when the controller tells it to | Several, added and removed as needed |
| Executor | A slot on a machine for one build at a time. An agent with 2 executors runs 2 builds at once | 2 per agent, 0 on the controller |
| Label | A tag on an agent, like linux or docker. A pipeline asks for a label, and any agent with it can run the build | linux |
| Queue | Builds waiting for a free executor with the right label | Hopefully short |
The controller used to be called the master, and you’ll still see that word in old guides. Today the controller’s own executors are called the built-in node.
Why builds must not run on the controller
| Reason | What goes wrong |
|---|---|
| Performance | A heavy build uses the CPU and memory that the scheduler and the web page need. Every other build and every user slows down |
| Security | A build on the controller runs as the same user as Jenkins. It can read JENKINS_HOME, where every credential and the key that decrypts them are kept. Anyone who can change a Jenkinsfile could read them all |
| Stability | A build that fills the disk fills the controller’s disk. Jenkins stops saving anything |
A fresh install gives the controller 2 executors so you can try things out. The first thing to do on a real Jenkins is set that to 0.
Where agents come from
| Kind | How it works | When it fits |
|---|---|---|
| Permanent agent | A server you run all the time, which connects to the controller | A small team with steady work. What you’ll set up in the lab |
| On demand, on EC2 | The Amazon EC2 plugin starts an EC2 instance when builds are waiting, and stops it when it has been idle for a while | Work that comes in bursts; you pay only while builds run |
| On demand, on Kubernetes | The Kubernetes plugin starts a pod for each build and deletes it afterwards | Teams that already run a Kubernetes cluster |
| A container per build | With the Docker Pipeline plugin, a pipeline says agent { docker { image 'node:22' } } and the agent runs that build inside the container | Each project needs different tools; the agent only needs Docker |
Whichever you choose, the idea is the same: agents are cattle, not pets. Any agent can be thrown away and replaced. Nothing important lives on one.
Running the controller on AWS
The controller is the one machine that isn’t disposable, so it gets the most care. QuickCart’s is built by Terraform, so it can be rebuilt the same way every time.
| Decision | QuickCart’s choice | Why |
|---|---|---|
| Where it runs | One EC2 instance | Jenkins runs one controller at a time; it doesn’t spread across servers |
| How Jenkins is installed | The official container image, jenkins/jenkins, pinned to an LTS version | Upgrading is changing one version number. LTS (long-term support) versions get fixes for longer |
| How people reach it | Session Manager, which is part of AWS Systems Manager. No SSH, no open ports | Access is controlled by IAM and logged. Nothing on the internet can reach the login page |
| How it gets AWS permissions | An instance role, not access keys | The permissions come with the machine and rotate automatically. There’s no key to leak |
| Where its data lives | JENKINS_HOME on an encrypted EBS volume | It survives a restart, and it can be snapshotted |
Later, when GitHub needs to reach Jenkins to announce a push, you put a load balancer with HTTPS in front of it. Continuous integration covers that.
Plugins
Almost everything Jenkins does comes from plugins. These are the ones QuickCart uses:
| Plugin | What it adds |
|---|---|
| Pipeline | Jenkinsfiles: pipelines written as code. Installs a group of plugins together |
| Git | Checking code out of a git repository |
| GitHub Branch Source | A job per branch and pull request, found automatically |
| Credentials and Credentials Binding | The credentials store, and withCredentials to use a secret in a build |
| Pipeline: AWS Steps | withAWS, to run steps with an AWS role |
| Amazon EC2 or Kubernetes | Agents started on demand |
| Docker Pipeline | agent { docker { image 'node:22' } } |
| Timestamper | A time on every line of a build log |
| AnsiColor | Coloured output from tools that print colour codes |
| Configuration as Code | The whole Jenkins setup in one YAML file |
Every plugin is code running inside the controller, with full access to it. Install only what you need, and keep them updated: most Jenkins security advisories are about plugins, not Jenkins itself.
Pin them. In the web page, Manage Jenkins → Plugins installs the newest version of each plugin. For a controller you can rebuild, list them in a file and bake them into your own image instead:
configuration-as-code
workflow-aggregator
git
github-branch-source
credentials-binding
pipeline-aws
timestamper
ansicolor
ec2
thinBackupFROM jenkins/jenkins:2.568.3-lts-jdk21
COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli --plugin-file /usr/share/jenkins/ref/plugins.txt| Line | What it does |
|---|---|
FROM jenkins/jenkins:2.568.3-lts-jdk21 | Start from the official image: this LTS version of Jenkins, running on Java 21 |
COPY … plugins.txt … | Put the list in the place the official image expects. --chown makes the jenkins user own it |
RUN jenkins-plugin-cli … | A tool in the official image that downloads every plugin in the list, and the plugins they depend on |
Each name in plugins.txt is a plugin’s ID, which is shown on its page in Manage Jenkins → Plugins. A plain ID takes the newest version. Once you have a set that works, write versions too, like git:5.7.0, so a rebuild gives exactly the same plugins. That version number is only an example; use the ones your working Jenkins shows.
The credentials store
Pipelines need secrets: a token for GitHub, a password for a registry. They go in the credentials store, never in a Jenkinsfile. A pipeline refers to a secret by its ID, and Jenkins hands over the value only while that step runs, hiding it in the log as ****.
| Kind | Holds | Example |
|---|---|---|
| Secret text | One secret value | A GitHub token for posting comments |
| Username with password | A pair | A registry login |
| SSH Username with private key | A private key | Checking out a repository over SSH |
| Secret file | A whole file | A kubeconfig for a cluster |
Every credential also has a scope. Global means pipelines can use it. System means only Jenkins itself can use it, for example to connect agents. You can also keep credentials inside a folder, so only the jobs in that folder can use them. That’s how you stop a test job from using the production deploy key.
Jenkins stores credentials encrypted in JENKINS_HOME. The key that decrypts them is in JENKINS_HOME/secrets/. That’s why builds must stay off the controller, and why a backup of JENKINS_HOME must be protected like a password.
For AWS, store nothing. An agent on EC2 gets permissions from its instance role. A pipeline that needs different permissions assumes a role with withAWS. Continuous delivery shows both. A long-lived AWS access key in Jenkins is the first thing an attacker looks for.
What to back up
Everything Jenkins knows lives in one folder, JENKINS_HOME. In the lab it’s /var/lib/jenkins-home on the instance, and /var/jenkins_home inside the container.
| In JENKINS_HOME | What it is | Lose it and |
|---|---|---|
config.xml and other *.xml | Jenkins’ own settings | You rebuild the setup by hand |
jobs/ | Every job’s settings and its build history | Build history and logs are gone |
credentials.xml | Stored credentials, encrypted | Every secret must be entered again |
secrets/ | The keys that decrypt them | The credentials above can never be decrypted |
plugins/ | Installed plugins | Reinstall them from your list |
workspace/ | Checked-out code for recent builds | Nothing. It’s rebuilt on the next build |
There are three ways to protect it. Most teams use the first two together.
| Way | What it does | Good for |
|---|---|---|
| EBS snapshots, scheduled with AWS Backup | A copy of the whole disk, every night. AWS Backup can pick volumes by tag and keep 30 days of copies | Getting everything back, fast |
| The ThinBackup plugin | Copies the settings, jobs and credentials to another folder on a schedule. That folder can then be copied to S3 | Going back to yesterday’s version of one job |
| Configuration as code | Jenkins’ settings in YAML, and every pipeline in a Jenkinsfile, all in git | Rebuilding a clean Jenkins from nothing, and seeing who changed what |
A backup you’ve never restored is a hope, not a backup. Try a restore before you need one.
Jenkins Configuration as Code
Clicking through Manage Jenkins leaves no record of what was changed or why. The Configuration as Code plugin, usually called JCasC, reads the whole setup from a YAML file when Jenkins starts:
jenkins:
systemMessage: "QuickCart CI. Set up by jenkins.yaml: changes made here are lost on restart."
numExecutors: 0
securityRealm:
local:
allowsSignup: false
users:
- id: "admin"
password: "${JENKINS_ADMIN_PASSWORD}"
authorizationStrategy:
loggedInUsersCanDoAnything:
allowAnonymousRead: false
unclassified:
location:
url: "https://jenkins.example.com/"| Setting | What it does |
|---|---|
systemMessage | A banner at the top of every page. Here it warns people that clicking changes won’t last |
numExecutors: 0 | No builds on the controller, as code |
securityRealm → local | Jenkins keeps its own list of users. Larger teams connect a company login instead |
allowsSignup: false | Nobody can create their own account |
password: "${JENKINS_ADMIN_PASSWORD}" | JCasC fills this in from an environment variable of that name, so the password isn’t in the file |
loggedInUsersCanDoAnything | Anyone logged in can do anything. Fine for a small team; larger teams use Matrix Authorization Strategy to give each person only what they need |
allowAnonymousRead: false | Visitors who aren’t logged in see nothing |
unclassified → location → url | The address Jenkins uses in links and in messages to GitHub |
Point Jenkins at the file with an environment variable, CASC_JENKINS_CONFIG=/path/to/jenkins.yaml. Manage Jenkins → Configuration as Code also has View Configuration, which shows the YAML for the Jenkins you already have. That’s the easiest way to start.
Security basics
| Do | Why |
|---|---|
| Keep the controller off the internet. Use Session Manager, a VPN, or a load balancer that allows only the sources you expect | Exposed Jenkins servers are scanned for and attacked constantly |
| Turn off sign-up and anonymous read | A Jenkins page can reveal code, logs and the names of your servers |
| Update Jenkins and plugins every month | Security fixes arrive constantly, mostly for plugins |
| Zero executors on the controller | Builds can’t read its secrets |
| Limit who can change jobs and Jenkinsfiles | Changing a pipeline means you can use every credential it can use |
| Treat Manage Jenkins → Script Console as root access | It runs any code on the controller. Only administrators can open it; keep that list short |
| Use roles for AWS, not stored keys | There’s nothing to steal |
Part 3: See it
Below is a Jenkins controller with one agent. Queue a few builds and watch who picks them up. Add an agent. Then give the controller 2 executors, like a fresh install, and queue two heavy builds.
Build queue
waiting for a free executor
- Empty
Controller
the brain: schedules, stores, serves the UI
- Web UI and APIresponsive
- Schedulerrunning
- JENKINS_HOMEjobs, history, credentials
No executors. Builds never run here.
Agents
the hands: each executor runs one build
Waiting 0 · Running 0 · Finished 0
What to notice: with 0 executors on the controller, a busy Jenkins only means a longer queue, and the fix is another agent. With executors on the controller, one heavy build can stall the scheduler, so free agents sit idle while everyone waits.
Part 4: Do it
Costs money — about $1.20 a day in us-east-1 while the instance runs: a t3.medium, its disk and its public IP address. Prices change; check the EC2 pricing page. Destroy it when you finish.
You’ll build a Jenkins controller on EC2 with Terraform, reach it without opening a single port, connect an agent, run a build that uses a stored secret, and take a backup. About 60 minutes.
You need Terraform 1.10 or later and a working AWS login in your terminal, as set up in Set up your computer. You also need the Session Manager plugin for the AWS CLI, which lets your computer open a tunnel to the instance. Install it the way that matches your system.
On macOS:
brew install --cask session-manager-plugin| Part | What it means |
|---|---|
brew install | install software with Homebrew |
--cask | this one is a packaged app rather than a command-line formula. Casks only exist on macOS |
session-manager-plugin | the plugin the AWS CLI uses for Session Manager |
On Linux, and on Windows inside Ubuntu (WSL), download AWS’s Ubuntu package:
curl -fsSL -o session-manager-plugin.deb https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb| Part | What it means |
|---|---|
curl -fsSL | download, failing on errors and following redirects |
-o session-manager-plugin.deb | save it to this file |
…/ubuntu_64bit/session-manager-plugin.deb | AWS’s package for Intel and AMD computers. On an ARM computer, use ubuntu_arm64 in place of ubuntu_64bit |
Then install it:
sudo dpkg -i session-manager-plugin.deb| Part | What it means |
|---|---|
sudo dpkg -i | install a downloaded Ubuntu package, as the administrator |
session-manager-plugin.deb | the file you just downloaded |
session-manager-plugin| Part | What it means |
|---|---|
session-manager-plugin | run the plugin on its own, just to check it’s installed |
The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.Write the Terraform for the controller
Make a folder called
quickcart-jenkinsand move into it:Terminalmkdir quickcart-jenkinsPart What it means mkdirmake a directory quickcart-jenkinsits name Terminalcd quickcart-jenkinsPart What it means cdchange directory, so every later command runs in here quickcart-jenkinsthe folder you just made Create
main.tf:main.tfterraform { required_version = ">= 1.10.0" required_providers { aws = { source = "hashicorp/aws" version = "~> 6.0" } } } provider "aws" { region = "us-east-1" } data "aws_vpc" "default" { default = true } data "aws_ami" "al2023" { most_recent = true owners = ["amazon"] filter { name = "name" values = ["al2023-ami-2023.*-x86_64"] } } resource "aws_security_group" "jenkins" { name = "quickcart-jenkins" description = "Jenkins controller: nothing in, everything out" vpc_id = data.aws_vpc.default.id } resource "aws_vpc_security_group_egress_rule" "all_out" { security_group_id = aws_security_group.jenkins.id description = "Pull images and plugins, reach Session Manager" ip_protocol = "-1" cidr_ipv4 = "0.0.0.0/0" } resource "aws_iam_role" "jenkins" { name = "quickcart-jenkins-controller" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Service = "ec2.amazonaws.com" } Action = "sts:AssumeRole" }] }) } resource "aws_iam_role_policy_attachment" "ssm" { role = aws_iam_role.jenkins.name policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" } resource "aws_iam_instance_profile" "jenkins" { name = "quickcart-jenkins-controller" role = aws_iam_role.jenkins.name } resource "aws_instance" "jenkins" { ami = data.aws_ami.al2023.id instance_type = "t3.medium" vpc_security_group_ids = [aws_security_group.jenkins.id] iam_instance_profile = aws_iam_instance_profile.jenkins.name associate_public_ip_address = true user_data_replace_on_change = true metadata_options { http_tokens = "required" http_put_response_hop_limit = 2 } root_block_device { volume_size = 30 volume_type = "gp3" encrypted = true } user_data = <<-EOT #!/bin/bash set -euo pipefail dnf install -y docker systemctl enable --now docker mkdir -p /var/lib/jenkins-home chown 1000:1000 /var/lib/jenkins-home docker network create jenkins docker run -d --name jenkins --restart unless-stopped --network jenkins -p 127.0.0.1:8080:8080 -v /var/lib/jenkins-home:/var/jenkins_home jenkins/jenkins:2.568.3-lts-jdk21 EOT tags = { Name = "quickcart-jenkins" Project = "quickcart" } } output "instance_id" { description = "Use this with Session Manager" value = aws_instance.jenkins.id }The settings, one block at a time
Setting What it does If you leave it out When you’d change it required_versionRefuse to run on Terraform older than 1.10 An old Terraform might fail later with a confusing error When you rely on a newer feature required_providers→awsUse the hashicorp/awsplugin, any 6.x versionTerraform picks the newest, which may one day bring breaking changes When you deliberately upgrade provider "aws"→regionBuild everything in us-east-1 Terraform uses the region from your AWS login settings To build somewhere else data "aws_vpc" "default"→default = trueLook up your account’s default VPC, which every account has in every region unless someone deleted it Required, to find the VPC In a real setup, use your own VPC with a private subnet data "aws_ami" "al2023"Look up a machine image to start from — — most_recent = trueIf several images match, take the newest Terraform fails when more than one matches Never. You want the latest security fixes owners = ["amazon"]Only accept images published by AWS itself Required. Without it, anyone’s image with a matching name could be picked Never for a server like this filter→name,valuesMatch Amazon Linux 2023 images for 64-bit Intel and AMD processors. *stands for the date part of the nameEvery image from Amazon would match Use arm64in the pattern for Graviton instances, which cost lessSetting What it does If you leave it out When you’d change it aws_security_group→nameThe firewall’s name in AWS Terraform makes up a name Never after creation; it forces a new one descriptionA note for people reading the console AWS writes Managed by TerraformNever after creation; it forces a new one vpc_idWhich VPC the firewall belongs to. It must match the instance’s It goes in the default VPC anyway, but saying so is clearer When you use your own VPC No ingress rules Nothing may connect in. Not SSH, not port 8080, nothing This is leaving it out, on purpose When a load balancer needs to reach Jenkins for webhooks. Then allow only the load balancer’s security group aws_vpc_security_group_egress_rule→security_group_idWhich firewall this rule belongs to Required — descriptionA note on the rule No note — ip_protocol = "-1"Every protocol and port Required Narrow it to port 443 if your network team asks cidr_ipv4 = "0.0.0.0/0"To anywhere. The instance must reach Docker Hub, the Jenkins plugin site and Session Manager Required Use VPC endpoints and a proxy in a locked-down network Setting What it does If you leave it out When you’d change it aws_iam_role→nameThe role the instance will use Terraform makes up a name — assume_role_policyWho may use the role: only the EC2 service, on behalf of an instance. jsonencodeturns the Terraform map into the JSON that IAM expectsRequired Never for an instance role aws_iam_role_policy_attachment→role,policy_arnGive the role AWS’s managed policy AmazonSSMManagedInstanceCore: the permissions the SSM agent needs so Session Manager works Session Manager can’t reach the instance, and you have no way in Add more policies here, such as pushing to ECR, when builds need them aws_iam_instance_profile→name,roleThe container that attaches a role to an EC2 instance. EC2 can’t use a role directly The instance can’t get the role Never Setting What it does If you leave it out When you’d change it amiStart from the Amazon Linux 2023 image found above. It comes with the SSM agent installed Required — instance_type = "t3.medium"2 vCPUs and 4 GiB of memory. Jenkins alone wants about 2 GiB Required Bigger if the web page gets slow with many jobs. Never to make room for builds vpc_security_group_idsUse the no-way-in firewall The VPC’s default security group, which may allow more — iam_instance_profileAttach the role The instance has no AWS permissions and Session Manager can’t reach it — associate_public_ip_address = trueGive it a public address, so it can reach the internet from the default VPC. The security group still lets nothing in In the default VPC it usually gets one anyway, but that depends on a subnet setting In a private subnet with a NAT gateway, set this to falseuser_data_replace_on_change = trueIf the start-up script changes, build a new instance, because the script only ever runs on the first boot Changing the script would do nothing to the running instance Never for a server built from its script metadata_options→http_tokens = "required"Only accept the newer, safer way of asking the instance for its role’s credentials, called IMDSv2 Older requests are also accepted, which some attacks rely on Never http_put_response_hop_limit = 2Let containers on the instance reach the instance’s credentials too. Each network hop uses one, and a container adds one With the default of 1, a container, such as an agent, can’t use the instance role Set it to 1 if no container should get the instance’s permissions root_block_device→volume_size = 30A 30 GiB disk. Build history and plugins grow over time 8 GiB, which fills up quickly Bigger if you keep many builds volume_type = "gp3"The current general-purpose SSD type, cheaper than the older gp2Depends on the image; may be gp2Rarely encrypted = trueEncrypt the disk and its snapshots Depends on your account’s default Never. JENKINS_HOMEholds secretstags→Name,ProjectNameis what the console shows in the instance listA nameless instance Add Backup = dailywhen AWS Backup picks instances by tagoutput "instance_id"Print the instance ID after apply. You’ll need it to open a tunnel You’d look it up in the console — The start-up script, line by line
user_dataruns once, as root, the first time the instance boots.<<-EOTandEOTmark the start and end of a block of text in Terraform.Line What it does #!/bin/bashRun this script with bash set -euo pipefailStop at the first command that fails, instead of carrying on half-set-up dnf install -y dockerInstall Docker with Amazon Linux’s package manager. -yanswers yessystemctl enable --now dockerStart Docker now, and on every boot mkdir -p /var/lib/jenkins-homeThe folder that will be JENKINS_HOME, on the instance’s diskchown 1000:1000 /var/lib/jenkins-homeThe official image runs Jenkins as user number 1000. It must own the folder to write to it docker network create jenkinsA private Docker network, so an agent container can reach the controller by the name jenkinsdocker run -d --name jenkinsStart the controller in the background, named jenkins--restart unless-stoppedStart it again after a crash or a reboot --network jenkinsJoin the private network -p 127.0.0.1:8080:8080Publish Jenkins’ web port only on the instance’s own loopback address. Even with a firewall mistake, nothing outside the instance could reach it. Session Manager connects from inside -v /var/lib/jenkins-home:/var/jenkins_homeKeep JENKINS_HOMEon the instance’s disk, so it survives the container being replacedjenkins/jenkins:2.568.3-lts-jdk21The official image, pinned to one LTS version. To upgrade, change this, and back up first Build it
Terminalterraform initPart What it means terraform initdownload the AWS provider and prepare the folder You should seeTerraform has been successfully initialized!Terminalterraform applyPart What it means terraform applyplan, show the plan, and build it once you type yesType
yeswhen asked.You should seePlan: 6 to add, 0 to change, 0 to destroy. Apply complete! Resources: 6 added, 0 changed, 0 destroyed. Outputs: instance_id = "i-0a1b2c3d4e5f6789"Your instance ID will be different. Keep it; the next steps call it
INSTANCE_ID. The instance needs about three minutes to install Docker and start Jenkins.Get the one-time admin password
Open a shell on the instance in your browser. Session Manager does this through AWS, with no SSH key and no open port.
In the AWS console: EC2Instancesquickcart-jenkinsConnect
Field Enter Tab Session Manager Choose Connect. A terminal opens in a new browser tab.
In the browser terminal, check Jenkins is running:
Terminalsudo docker psPart What it means sudorun as the administrator; Session Manager logs you in as a normal user docker pslist running containers You should seeCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3f2a9c1d7b4e jenkins/jenkins:2.568.3-lts-jdk21 "/usr/bin/tini -- /u…" 2 minutes ago Up 2 minutes 127.0.0.1:8080->8080/tcp, 50000/tcp jenkinsYour ID and times will differ. Then print the password Jenkins created on its first start:
Terminalsudo cat /var/lib/jenkins-home/secrets/initialAdminPasswordPart What it means sudo catprint a file, as the administrator /var/lib/jenkins-home/…JENKINS_HOMEon the instance’s disksecrets/initialAdminPassworda one-time password, used only to unlock the setup page You should see3b9e5c07a1d24f6e8c2b0d9a7f41e635Yours is different. Copy it. Leave this browser tab open; you’ll use it again.
Open a tunnel to Jenkins from your computer
The console can give you a shell, but it has no button for forwarding a port to your own browser, so this step uses the AWS CLI. In a terminal on your computer:
Terminalaws ssm start-session --target INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters portNumber=8080,localPortNumber=8080Part What it means aws ssm start-sessionstart a Session Manager session --target INSTANCE_IDwhich instance. Replace it with the instance_idTerraform printed--document-name AWS-StartPortForwardingSessionthe kind of session: forward a port, rather than open a shell --parameters portNumber=8080the port on the instance, where Jenkins listens localPortNumber=8080the port on your computer You should seeStarting session with SessionId: quickcart-admin-0f1e2d3c4b5a69788 Port 8080 opened for sessionId quickcart-admin-0f1e2d3c4b5a69788. Waiting for connections...Leave it running. Open
http://localhost:8080in your browser. Your computer passes it through the tunnel, AWS passes it to the SSM agent on the instance, and the agent connects to Jenkins on the instance’s loopback address.Finish the setup in the browser
Go through the setup pages:
Page Do this Unlock Jenkins Paste the one-time password. Choose Continue Customize Jenkins Install suggested plugins. It takes a few minutes Create First Admin User Choose a username and a long password. Store them in your password manager Instance Configuration Leave Jenkins URL as http://localhost:8080/. Choose Save and FinishChoose Start using Jenkins.
You should seeThe Jenkins dashboard, with Welcome to Jenkins! in the middle.
Take the builds off the controller
Go to Manage Jenkins → Nodes → Built-In Node → Configure.
Field Enter Number of executors 0Choose Save.
You should seeBack on the dashboard, the Build Executor Status panel on the left no longer lists any executors for the built-in node.
Store a secret
Go to Manage Jenkins → Credentials → System → Global credentials (unrestricted) → Add Credentials. Choose the kind Secret text, then:
Field Enter Scope Global Secret not-a-real-token-123ID quickcart-demo-tokenDescription Demo token for the Jenkins lessonChoose Create.
You should seequickcart-demo-tokenin the list. The value itself is never shown again. Anyone with permission can replace it, but nobody can read it back in the web page.Create a job and watch it wait
From the dashboard, choose New Item.
Field Enter Enter an item name hello-agentType Pipeline Choose OK. Scroll to Pipeline, leave Definition as Pipeline script, and paste this into Script:
Pipeline scriptpipeline { agent { label 'linux' } stages { stage('Where am I?') { steps { sh 'hostname' sh 'whoami' } } stage('Use a secret') { steps { withCredentials([string(credentialsId: 'quickcart-demo-token', variable: 'DEMO_TOKEN')]) { sh 'echo "The token is $DEMO_TOKEN"' } } } } }Line What it means agent { label 'linux' }run on any agent labelled linuxstage('Where am I?')a named step of the pipeline. hostnameandwhoamishow which machine and which user ran itwithCredentials([...])fetch a credential from the store, only for the steps inside the braces string(credentialsId: 'quickcart-demo-token', variable: 'DEMO_TOKEN')a Secret text credential, put into an environment variable called DEMO_TOKENsh 'echo "The token is $DEMO_TOKEN"'print it, to see what Jenkins does with it. The single quotes matter; see the mistake below The Jenkinsfile explains every part of a pipeline. Choose Save, then Build Now.
You should seeUnder Builds, build #1 waits with a clock icon. Hold the pointer over it, or open it: Jenkins says there are no nodes with the label
linux. There is no agent yet, and the controller now refuses to run builds. That’s correct.Connect an agent
Go to Manage Jenkins → Nodes → New Node.
Field Enter Node name agent-1Type Permanent Agent Choose Create, then:
Field Enter Number of executors 2Remote root directory /home/jenkins/agentLabels linuxUsage Use this node as much as possible Launch method Launch agent by connecting it to the controller Choose Save, then open agent-1 in the list. Jenkins shows the commands to start the agent. Copy the long secret from them: the 64-character value after
-secret. The next command calls itAGENT_SECRET.In a real setup, the agent is a separate EC2 instance. To keep this lab to one server, you’ll run it as a second container on the same instance. Go back to the Session Manager browser tab and run:
Terminalsudo docker run -d --name agent-1 --init --network jenkins -e JENKINS_URL=http://jenkins:8080/ -e JENKINS_AGENT_NAME=agent-1 -e JENKINS_SECRET=AGENT_SECRET -e JENKINS_WEB_SOCKET=true -e JENKINS_AGENT_WORKDIR=/home/jenkins/agent jenkins/inbound-agentPart What it means sudo docker run -dstart a container in the background --name agent-1call the container agent-1--initadd a tiny process that tidies up finished processes. Recommended for agent containers --network jenkinsjoin the same private network as the controller -e JENKINS_URL=http://jenkins:8080/where the controller is. On this network, jenkinsis the controller container’s name-e JENKINS_AGENT_NAME=agent-1which node in Jenkins this agent is -e JENKINS_SECRET=AGENT_SECRETproves it’s allowed to be that node. Replace it with the secret you copied -e JENKINS_WEB_SOCKET=trueconnect over the same port as the web page, so no extra port is needed -e JENKINS_AGENT_WORKDIR=/home/jenkins/agentwhere the agent keeps its files jenkins/inbound-agentthe official image for an agent that connects to its controller You should seeWithin a few seconds, build #1 starts by itself. Its Console Output says
Running on agent-1, then prints the agent container’s hostname, the userjenkins, and then:The console output includesThe token is ****Jenkins replaced the secret with
****in the log, because it knows the value it handed over.Back up JENKINS_HOME
JENKINS_HOMEis on the instance’s disk, so a snapshot of that disk is a complete backup.In the AWS console: EC2SnapshotsCreate snapshot
Field Enter Resource type Instance Instance ID the quickcart-jenkinsinstanceDescription quickcart-jenkins backupChoose Create snapshot.
You should seeA snapshot in the list with Snapshot status: Pending, then Completed after a few minutes.
To restore it, you’d create a volume from this snapshot and start a new instance with it. For a real controller, don’t take snapshots by hand: in AWS Backup, create a backup plan with a daily rule that keeps 30 days, and assign it to resources with a tag such as
Backup = daily. And once a quarter, restore one and check Jenkins starts from it.Clean-up step: Clean up
In the terminal running the tunnel, press Ctrl+C to close it.
Delete the snapshot first. Terraform didn’t create it, so
destroywon’t remove it.In the AWS console: EC2Snapshotsquickcart-jenkins backupActionsDelete snapshot
Then, in the
quickcart-jenkinsfolder:Terminalterraform destroyPart What it means terraform destroyremove everything in the state, once you type yesYou should seeDestroy complete! Resources: 6 destroyed.You should seeIn EC2 → Instances,
quickcart-jenkinsshows Terminated. It disappears from the list after about an hour.
Part 5: Check yourself
QuickCart's Jenkins has 2 executors on the controller and no agents. A large image build makes the web UI freeze for everyone. What is the right fix?
Why does a build that runs on the controller put secrets at risk?
Why does the Terraform in this lesson give the Jenkins instance no inbound security group rules at all?
The Jenkins instance is terminated by accident. Which backup gets QuickCart back fastest, with its jobs, build history and credentials?
A pipeline prints a secret with sh "echo ${TOKEN}" in double quotes. What's wrong?
Part 6: Cheat sheet
| Thing | Remember |
|---|---|
| Controller | Schedules, stores, serves the UI. 0 executors |
| Agent | Runs builds. Disposable. Picked by label |
| Agents on demand | Amazon EC2 plugin or Kubernetes plugin |
| Reaching the controller | Session Manager: aws ssm start-session --target INSTANCE_ID --document-name AWS-StartPortForwardingSession --parameters portNumber=8080,localPortNumber=8080 |
| AWS permissions | Instance role, or withAWS(role: …). Never stored keys |
| Secrets | Credentials store, used with withCredentials, in single quotes |
| Credential scope | Global for pipelines, System for Jenkins itself, folders to fence them off |
JENKINS_HOME | Everything. secrets/ decrypts credentials.xml |
| Backups | Nightly EBS snapshots with AWS Backup; ThinBackup for settings; restore one every quarter |
| Setup as code | jenkins.yaml with the Configuration as Code plugin; plugins.txt in your own image |
| Security | Not on the internet, no anonymous read, update monthly, short admin list |