Laptop to Production

Kubernetes and EKS

Pods, deployments, and the load balancer controller.

Time
2 hours
Level
Advanced

Part 1: The problem

QuickCart, the Kubernetes demo

The team can’t agree about Kubernetes. Kofi says it’s the standard; Priya says ECS is enough. Nobody has actually run QuickCart on it, so the argument goes in circles.

So Kofi builds a small EKS cluster one afternoon, runs the order service on it, and shows the team. The demo goes well. Everyone moves on.

Two months later, Ana, reading the AWS bill, finds a cluster nobody remembers, two servers nobody uses, and a load balancer pointing at nothing. The demo was still running.

QuickCart needs two things from this lesson: to understand what Kubernetes really is, so the argument can be settled with facts, and a way to try EKS that is built and torn down completely in one sitting.

Part 2: The idea

Kubernetes does the same job as ECS: it runs your containers, restarts them when they fail, keeps a set number running, and replaces them one by one when you release. It’s open source and runs anywhere: on every major cloud, and on your own servers. EKS is AWS’s managed Kubernetes.

The big idea is how you talk to it. You don’t tell Kubernetes what to do. You tell it what you want, in a file, and it keeps making that true.

Key point: You don’t tell Kubernetes what to do. You tell it what you want, and it keeps making that true.

The reconcile loop

Inside Kubernetes, small programs called controllers each watch one kind of object. Each one runs the same loop, over and over, for ever:

StepWhat happensExample
ObserveLook at what’s really running1 pod with the label app: order-service
CompareCompare it with the desired state you wroteThe file says 2. One is missing
ActChange reality to close the gapCreate one pod

That loop is where self-healing comes from. A pod crashes, a node disappears, someone deletes a pod by mistake: the numbers stop matching, and the controller acts. Nobody has to be awake.

The objects, compared with ECS

KubernetesWhat it isClosest thing on ECS
PodOne running copy of your app: one or more containers that share a network addressTask
DeploymentSays “run N pods of this image” and handles releasesService
ReplicaSetCreated by a Deployment, one per version. Keeps exactly N pods of that version— ECS does this inside the service
ServiceA fixed name and address in front of pods that come and goThe target group, roughly
IngressA request for a load balancer, with routing rulesLoad balancer and listener
NodeA server that pods run onHidden on Fargate
NamespaceA named folder for objects, like default or kube-systemThe cluster, roughly

A Deployment doesn’t create pods itself. It creates a ReplicaSet for each version of your app, and the ReplicaSet creates the pods. During a release, the Deployment grows the new ReplicaSet and shrinks the old one. That’s also how kubectl rollout undo is instant: the old ReplicaSet is still there, just scaled to zero.

Probes: is it alive, and is it ready?

Kubernetes checks each container with probes, small requests it makes on a schedule.

ProbeThe questionIf it fails
Readiness probeCan this pod take traffic right now?The pod is taken out of the Service. It keeps running, and gets traffic again when the probe passes
Liveness probeIs this container stuck beyond repair?The container is restarted
Startup probe (optional)Has a slow app finished starting?The other probes wait until it passes

This is a real difference from ECS. On ECS, a task failing its load balancer health check is replaced. On Kubernetes, a pod failing its readiness probe is only taken out of traffic, and a failing liveness probe restarts the container in the same pod.

What EKS runs, and what you run

A Kubernetes cluster has two halves.

HalfWhat it doesOn EKS, who runs it
Control planeThe API you talk to, the controllers, the scheduler, and the database of desired stateAWS, across several zones. About $0.10 an hour per cluster on standard support, roughly $73 a month
Data planeThe nodes, where pods actually runYou, in one of the ways below

Each Kubernetes version gets about 14 months of standard support on EKS. After that it moves to extended support, which costs about six times more for the control plane, until you upgrade. So an EKS cluster isn’t set up once: it needs upgrading, with its nodes and add-ons, for as long as it exists. Prices change; check the EKS pricing page.

Where pods runWhat it means
Managed node groupEC2 instances that AWS launches and helps you patch and replace. You pay for the instances. This lesson uses one
FargateNo nodes to see: each pod gets its own small Fargate machine. Some features, like daemon sets, don’t work
Auto ModeAWS also runs the nodes and core add-ons for you, for an extra fee on top of the instances

Every pod gets its own IP address from your subnet, and each node size has a limit. A t3.small holds about 11 pods, including Kubernetes’ own system pods, however much CPU is free.

Getting traffic in

A Service gives pods a stable name inside the cluster, but nothing outside can reach it. To get an AWS load balancer, you install the AWS Load Balancer Controller into the cluster. It watches for Ingress objects and builds a real Application Load Balancer for each one.

To build load balancers, the controller needs AWS permissions. You give them to that controller alone, not to every pod on the node, with IAM roles for service accounts (IRSA): a Kubernetes identity linked to an IAM role. It’s the same idea as ECS’s task role. EKS also has a newer way to do this, called Pod Identity; the idea is the same.

Releasing without dropped requests

The settings from Releases without downtime have Kubernetes equivalents:

ECSKubernetesQuickCart uses
Min running tasks 100%maxUnavailable0: never go below 2 ready pods
Max running tasks 200%maxSurge1: one extra pod at a time
Health checkReadiness probe/health
Circuit breaker with rollbackNo automatic rollback. The rollout stops and waitskubectl rollout undo

Kubernetes needs two extra settings that ECS handles for you. When a pod is removed, Kubernetes takes it out of the Service and sends it SIGTERM at the same moment. The load balancer takes a few seconds to catch up, and without help, requests arrive at a pod that has already gone.

SettingWhat it fixes
Pod readiness gateA new pod only counts as ready once the load balancer says it’s healthy, not just its own probe. So old pods aren’t removed too early
preStop sleep of 15 secondsThe pod keeps serving for 15 seconds after it’s told to stop, while the load balancer stops sending it traffic. Then SIGTERM arrives, and gunicorn finishes and exits

terminationGracePeriodSeconds is Kubernetes’ stop timeout. It covers the preStop wait and the app’s shutdown, so QuickCart sets it to 40: 15 for the sleep, 20 for gunicorn’s graceful timeout, and 5 to spare.

Part 3: See it

Here is the order service’s Deployment asking for 2 pods, on a cluster with two nodes. Delete a pod and watch the loop put it back. Then change the number of replicas, and stop node B.

Replicas you ask for
Desired state, from order-service.yaml
replicas: 2
Running and ready: 2 of 2
  1. Observecount the pods
  2. Comparewith replicas
  3. Actcreate or delete
Node AReady
  • order-service-7d4b9c8f6-x2kqpRunning, ready
Node BReady
  • order-service-7d4b9c8f6-m8zrtRunning, ready
Desired 2, pods 2Everything matches, so the controller does nothing. Choose Delete a pod and watch what it does about it.

What to notice: you never told Kubernetes to create anything. Every pod it made came from the same three steps: observe, compare, act. And the replacement always gets a new name. Pods are disposable: nothing should depend on one particular pod still existing.

Part 4: Do it

You’ll create an EKS cluster, install the load balancer controller, run QuickCart’s order service on it, watch Kubernetes replace a deleted pod, release v2 under load and roll it back, and then delete everything.

You need the quickcart-orders repository in ECR with the v1 and v2 images from Releases without downtime, your load.py, and a working AWS login in your terminal as an IAM user or role, not root. Set up your computer covers the login.

Why eksctl, not the console

You can create an EKS cluster in the console, but it takes two IAM roles you create first, a cluster form, a wait, a separate node group form, and then the terminal anyway, because kubectl and Helm only run there. eksctl is AWS’s recommended command-line tool for EKS. One command builds the control plane, a managed node group, their IAM roles and the networking, and connects kubectl to the new cluster. It uses CloudFormation underneath, so one more command deletes all of it again. This lab uses eksctl, and still uses the console wherever it has an equivalent.

  1. Install the tools

    Terminal
    brew install eksctl kubectl helm
    PartWhat it means
    brew installinstall programs on your computer with Homebrew
    eksctlcreates and deletes EKS clusters
    kubectltalks to any Kubernetes cluster: creates, lists and deletes objects
    helminstalls packaged add-ons, called charts, into a cluster

    Check that each one works:

    Terminal
    kubectl version --client
    PartWhat it means
    kubectl versionprint the version
    --clientonly this program’s version. There’s no cluster to ask yet
    You should see

    A line starting Client Version: v1., followed by a version number. eksctl version and helm version print their versions too. Yours will be newer than anything written here.

  2. Check who you're logged in as

    The identity that creates the cluster is the one that can use it straight away. Make sure it’s the one you’ll keep using:

    Terminal
    aws sts get-caller-identity
    PartWhat it means
    aws sts get-caller-identityask AWS who this terminal is logged in as
    You should see
    {
        "UserId": "AIDAEXAMPLEUSERID",
        "Account": "111122223333",
        "Arn": "arn:aws:iam::111122223333:user/quickcart-admin"
    }

    Your IDs and name will differ. The Arn must not end in :root.

  3. Find and tag two subnets

    In the AWS console: VPCSubnets

    Find the default VPC’s subnets in us-east-1a and us-east-1b: their VPC column shows the default VPC. Copy both Subnet IDs, and the VPC ID. They look like subnet- and vpc- followed by letters and numbers.

    Then tag each of the two subnets, so the load balancer controller knows it may put an internet-facing load balancer there:

    In the AWS console: VPCSubnetsyour us-east-1a subnetTagsManage tagsAdd new tag

    FieldEnter
    Keykubernetes.io/role/elb
    Value1

    Choose Save, then do the same for the us-east-1b subnet.

  4. Create the cluster

    Put your two subnet IDs in place of SUBNET_A and SUBNET_B, with a comma and no space between them. The \ at the end of a line means the command carries on to the next line.

    Terminal
    eksctl create cluster \
      --name quickcart-eks \
      --region us-east-1 \
      --vpc-public-subnets SUBNET_A,SUBNET_B \
      --nodegroup-name quickcart-nodes \
      --node-type t3.small \
      --nodes 2 \
      --managed
    PartWhat it means
    eksctl create clusterbuild a new EKS cluster, with everything it needs
    --name quickcart-eksthe cluster’s name
    --region us-east-1the region to build it in
    --vpc-public-subnets SUBNET_A,SUBNET_Buse these two existing subnets in your default VPC, instead of building a new VPC
    --nodegroup-name quickcart-nodesthe name of the group of servers
    --node-type t3.smallthe size of each server: 2 vCPUs, 2 GB of memory, about 11 pods
    --nodes 2two servers, so there’s one in each zone
    --manageda managed node group: AWS launches, and helps you update, the servers

    There’s no --version, on purpose: eksctl picks a current version on standard support.

    You should see

    Many lines over about 15 to 20 minutes, as eksctl builds the control plane and then the nodes. It ends with a line like EKS cluster "quickcart-eks" in "us-east-1" region is ready. It has also saved the cluster’s details for kubectl, so the next command just works.

    Check the nodes:

    Terminal
    kubectl get nodes
    PartWhat it means
    kubectl get nodeslist the servers in the cluster
    You should see
    NAME                            STATUS   ROLES    AGE   VERSION
    ip-172-31-17-201.ec2.internal   Ready    <none>   3m    v1.33.4-eks-a1b2c3d
    ip-172-31-90-12.ec2.internal    Ready    <none>   3m    v1.33.4-eks-a1b2c3d

    Your names, ages and versions will differ. Both must say Ready. In EC2 → Instances there are now two new instances: they’re yours, and on your bill.

  5. Let one pod have its own IAM role

    The load balancer controller needs AWS permissions. This command switches on the feature that lets a single Kubernetes identity use an IAM role:

    Terminal
    eksctl utils associate-iam-oidc-provider --cluster quickcart-eks --region us-east-1 --approve
    PartWhat it means
    eksctl utils associate-iam-oidc-providerregister the cluster with IAM as an identity provider, so IAM can trust identities from inside it
    --cluster quickcart-ekswhich cluster
    --region us-east-1where the cluster is
    --approveactually do it. Without this, eksctl only shows what it would do
    You should see

    A line saying the IAM OIDC provider was created for the cluster. In IAM → Identity providers, a new provider whose address mentions oidc.eks.us-east-1.

  6. Create the controller's IAM policy

    The controller’s permissions are published as a JSON file called iam_policy.json, linked from the AWS Load Balancer Controller installation guide in the Amazon EKS documentation. It’s long, and AWS updates it with each controller release, so always take the current file from the guide rather than an old copy. Download it, and open it in a text editor.

    In the AWS console: IAMPoliciesCreate policyJSON

    Delete everything in the editor, then paste in the whole contents of iam_policy.json. Choose Next.

    FieldEnter
    Policy nameQuickCartLBControllerPolicy

    Choose Create policy.

    You should see

    The policy in the list. Open it and copy its ARN: arn:aws:iam::111122223333:policy/QuickCartLBControllerPolicy, with your account ID.

  7. Give the controller that policy

    This step and the next have no console equivalent. Linking an IAM role to an identity inside Kubernetes takes a role with a special trust policy plus a Kubernetes object; eksctl does both in one command. Put your account ID in the ARN:

    Terminal
    eksctl create iamserviceaccount \
      --cluster quickcart-eks \
      --region us-east-1 \
      --namespace kube-system \
      --name aws-load-balancer-controller \
      --role-name QuickCartLBControllerRole \
      --attach-policy-arn arn:aws:iam::111122223333:policy/QuickCartLBControllerPolicy \
      --approve
    PartWhat it means
    eksctl create iamserviceaccountcreate a Kubernetes service account (an identity for pods) and an IAM role linked to it
    --namespace kube-systemput it with Kubernetes’ own system programs
    --name aws-load-balancer-controllerthe service account’s name. The controller will run as this identity
    --role-name QuickCartLBControllerRolethe IAM role’s name, so you can find it later
    --attach-policy-arngive the role the policy you just created
    --approveactually do it
    You should see

    About a minute of output, including a CloudFormation stack being created, and a line saying the service account was created.

  8. Install the controller with Helm

    First tell Helm where AWS publishes its EKS charts. The repository address is in the same installation guide, in its Helm section; put it in place of CHART_REPO_URL:

    Terminal
    helm repo add eks CHART_REPO_URL
    PartWhat it means
    helm repo addremember a chart repository, so you can install from it
    eksthe short name you’ll use for it
    CHART_REPO_URLreplace this with the eks-charts repository address from the installation guide
    You should see
    "eks" has been added to your repositories

    Now install the controller. Put your VPC ID in place of VPC_ID:

    Terminal
    helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
      --namespace kube-system \
      --set clusterName=quickcart-eks \
      --set serviceAccount.create=false \
      --set serviceAccount.name=aws-load-balancer-controller \
      --set region=us-east-1 \
      --set vpcId=VPC_ID
    PartWhat it means
    helm install aws-load-balancer-controllerinstall a chart, and call this installation aws-load-balancer-controller
    eks/aws-load-balancer-controllerthe chart: from the eks repository, the one called aws-load-balancer-controller
    --namespace kube-systeminstall it next to the service account you made
    --set clusterName=quickcart-ekstell it which cluster it’s in, so it can name and tag what it builds
    --set serviceAccount.create=falsedon’t make a new identity
    --set serviceAccount.name=aws-load-balancer-controlleruse the one eksctl made, with the IAM role
    --set region=us-east-1 and --set vpcId=VPC_IDtell it where it is. Without these, it tries to look them up from the node, which can fail
    You should see

    A few lines ending in STATUS: deployed, and a note that the controller is installed.

    Check it’s running:

    Terminal
    kubectl get deployment --namespace kube-system aws-load-balancer-controller
    PartWhat it means
    kubectl get deploymentshow a Deployment and how many of its pods are ready
    --namespace kube-systemlook in the kube-system namespace, not default
    aws-load-balancer-controllerwhich Deployment
    You should see
    NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
    aws-load-balancer-controller   2/2     2            2           70s

    If it shows 0/2, wait a minute and run it again. Count what you just did to connect a load balancer: an identity provider, a policy, a service account, a chart. On ECS it was one section of the service form. That difference belongs in the ECS or EKS decision.

  9. Turn on readiness gates

    Terminal
    kubectl label namespace default elbv2.k8s.aws/pod-readiness-gate-inject=enabled
    PartWhat it means
    kubectl labeladd a label to an object
    namespace defaultthe object: the default namespace, where your app will run
    elbv2.k8s.aws/pod-readiness-gate-inject=enabledthe label the controller looks for. Pods created in this namespace from now on get a readiness gate
    You should see
    namespace/default labeled

    From now on, a new pod here only counts as ready once the load balancer reports it healthy. Label the namespace before creating the pods: pods that already exist don’t get the gate.

  10. Describe the app

    In your order-service folder, create order-service.yaml. It describes three objects, separated by --- lines. Put your account ID in the image address.

    order-service.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: order-service
      labels:
        app: order-service
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: order-service
      strategy:
        type: RollingUpdate
        rollingUpdate:
          maxUnavailable: 0
          maxSurge: 1
      template:
        metadata:
          labels:
            app: order-service
        spec:
          terminationGracePeriodSeconds: 40
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
          containers:
            - name: app
              image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v1
              ports:
                - containerPort: 8080
              readinessProbe:
                httpGet:
                  path: /health
                  port: 8080
                periodSeconds: 5
                failureThreshold: 2
              livenessProbe:
                httpGet:
                  path: /health
                  port: 8080
                initialDelaySeconds: 15
                periodSeconds: 10
                failureThreshold: 3
              resources:
                requests:
                  cpu: 100m
                  memory: 128Mi
                limits:
                  memory: 256Mi
              securityContext:
                allowPrivilegeEscalation: false
              lifecycle:
                preStop:
                  exec:
                    command: ["sleep", "15"]
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: order-service
    spec:
      selector:
        app: order-service
      ports:
        - port: 80
          targetPort: 8080
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: order-service
      annotations:
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/healthcheck-path: /health
        alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
    spec:
      ingressClassName: alb
      rules:
        - http:
            paths:
              - path: /
                pathType: Prefix
                backend:
                  service:
                    name: order-service
                    port:
                      number: 80

    The Deployment

    SettingWhat it doesIf you leave it outWhen you’d change it
    apiVersion: apps/v1, kind: DeploymentWhich kind of object this isRequiredNever
    metadata.nameThe Deployment’s name, used by kubectl commandsRequiredNever, once created
    replicas: 2The desired state: 2 pods1 podTo scale by hand, or let an autoscaler set it
    selector.matchLabelsWhich pods belong to this Deployment: those labelled app: order-serviceRequiredNever. It can’t be changed after creation
    strategy.rollingUpdate.maxUnavailable: 0During a release, never have fewer than 2 ready pods25% of the replicas may be missingHigher, to release faster when spare capacity doesn’t matter
    strategy.rollingUpdate.maxSurge: 1During a release, run at most 1 extra pod25% extra, rounded upHigher, to release faster when you have spare node capacity
    template.metadata.labelsLabels on every pod. Must match the selectorThe Deployment is rejectedAdd labels for version or team
    terminationGracePeriodSeconds: 40Kubernetes’ stop timeout: 15 s preStop + 20 s gunicorn + 5 spare30 seconds: gunicorn could be killed mid-requestIf your requests or shutdown take longer
    securityContext.runAsNonRoot: trueRefuse to start the container as rootNothing stops a root containerNever
    securityContext.runAsUser: 1001Run as user 1001, the appuser from the DockerfileUses the image’s USEROnly if the image’s user changes
    containers[].name: appThe container’s name, used by kubectl set imageRequiredNever
    imageThe image to run, from ECR, by tagRequiredEvery release
    ports.containerPort: 8080The port the app listens on. Mostly documentation, like EXPOSEThe app still works; tools that read it can’tIf the app’s port changes
    readinessProbeAsk /health every 5 s; 2 failures in a row takes the pod out of trafficThe pod gets traffic as soon as it starts, before the app is readyTighter for fast apps, looser for slow ones
    livenessProbeAsk /health every 10 s after 15 s; 3 failures in a row restart the containerA stuck app is never restartedKeep it lenient. See the mistake in the idea
    resources.requestsWhat the scheduler reserves on a node: a tenth of a CPU and 128 MBThe pod may land on a node with no room, and be squeezedMeasure real use, then set it a little above
    resources.limits.memory: 256MiThe most memory the container may use. Above it, it’s killed, shown as OOMKilledIt can use all of a node’s memory, and starve its neighboursWhen the app really needs more
    allowPrivilegeEscalation: falseThe process can never gain more privileges than it starts withAllowedNever
    lifecycle.preStop sleep 15Keep serving for 15 s after being told to stop, while the load balancer catches upRequests arrive at a pod that has already goneMatch it to how long your load balancer takes to deregister

    The Service and the Ingress

    SettingWhat it doesIf you leave it outWhen you’d change it
    Service selector: app: order-serviceSend traffic to pods with this label, as long as they’re readyThe Service has no pods behind itNever, while the labels stay the same
    Service port: 80, targetPort: 8080Inside the cluster, order-service:80 reaches the pods on 8080RequiredIf the app’s port changes
    ingressClassName: albHand this Ingress to the AWS Load Balancer ControllerNo controller claims it, and no load balancer is builtNever, on EKS with this controller
    alb.ingress.kubernetes.io/scheme: internet-facingA public load balancerinternal: reachable only inside the VPCFor services only other services call
    alb.ingress.kubernetes.io/target-type: ipRegister pod IPs directly, like Fargate tasksinstance: traffic goes via a port on each node, an extra hopRarely
    alb.ingress.kubernetes.io/healthcheck-path: /healthThe ALB’s health check path/, which works here, but isn’t a real readiness checkIf the health path changes
    target-group-attributes: deregistration_delay.timeout_seconds=30The drain, as on ECS300 seconds: slow releasesLonger than your slowest request
    rules with path: / and pathType: PrefixSend every path to the order-service Service on port 80RequiredTo route different paths to different Services

    This Ingress listens on HTTP port 80 only. For HTTPS you’d add a certificate from Certificate Manager with the alb.ingress.kubernetes.io/certificate-arn annotation and a listener on 443, as in Domain names and HTTPS.

  11. Start it

    Terminal
    kubectl apply -f order-service.yaml
    PartWhat it means
    kubectl applymake the cluster match a file: create what’s new, update what changed
    -f order-service.yamlthe file to use
    You should see
    deployment.apps/order-service created
    service/order-service created
    ingress.networking.k8s.io/order-service created
    Terminal
    kubectl get pods
    PartWhat it means
    kubectl get podslist the pods in the default namespace
    You should see
    NAME                             READY   STATUS    RESTARTS   AGE
    order-service-7d4b9c8f6-x2kqp    1/1     Running   0          40s
    order-service-7d4b9c8f6-m8zrt    1/1     Running   0          40s

    The part in the middle of each name comes from the ReplicaSet; the end is random. Yours differ.

    Now wait for the load balancer:

    Terminal
    kubectl get ingress
    PartWhat it means
    kubectl get ingresslist Ingresses, with the address of the load balancer built for each
    You should see
    NAME            CLASS   HOSTS   ADDRESS                                                                  PORTS   AGE
    order-service   alb     *       k8s-default-orderser-1a2b3c4d5e-1234567890.us-east-1.elb.amazonaws.com   80      3m

    The ADDRESS appears after two or three minutes; until then, it’s empty. In EC2 → Load balancers, a new load balancer starting with k8s-default- exists. Your YAML file made a real AWS resource.

  12. Reach it, and delete a pod

    Copy the ADDRESS from the last command:

    Terminal
    curl INGRESS_ADDRESS/
    PartWhat it means
    curlfetch the page and print it
    INGRESS_ADDRESSreplace this with the address from kubectl get ingress
    You should see
    {"served_by":"order-service-7d4b9c8f6-x2kqp","service":"order-service","version":"v1"}

    A pod’s hostname is its name, so served_by tells you which pod answered. Run it a few times: it switches between the two.

    Now delete one of them, using a name from your kubectl get pods:

    Terminal
    kubectl delete pod POD_NAME
    PartWhat it means
    kubectl delete poddelete one pod
    POD_NAMEreplace this with one of your pod names
    Terminal
    kubectl get pods
    PartWhat it means
    kubectl get podslist the pods again, to see what Kubernetes did
    You should see

    Two pods again, 1/1 Running. One has the name you didn’t delete; the other is new, with an AGE of a few seconds. You changed nothing in the file. The reconcile loop saw 1 where you’d asked for 2, and acted.

  13. Release v2 under load, then roll back

    Window 1: start the load test, with the Ingress address:

    Terminal
    python3 load.py http://INGRESS_ADDRESS/ 240
    PartWhat it means
    python3 load.pythe load test from Releases without downtime
    http://INGRESS_ADDRESS/the EKS load balancer this time
    240four minutes

    Window 2: change the image. Put your account ID in:

    Terminal
    kubectl set image deployment/order-service app=111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v2
    PartWhat it means
    kubectl set imagechange the image of a container in an object
    deployment/order-servicewhich object
    app=which container: the one named app
    the restthe new image: the v2 tag
    You should see
    deployment.apps/order-service image updated

    Watch the release:

    Terminal
    kubectl rollout status deployment/order-service
    PartWhat it means
    kubectl rollout statuswait, printing progress, until the release finishes or fails
    You should see

    Lines like Waiting for deployment "order-service" rollout to finish, one pod at a time because of maxSurge: 1, then deployment "order-service" successfully rolled out. When window 1 finishes, write down its totals. The target is failed: 0.

    Now go back to v1, with one command:

    Terminal
    kubectl rollout undo deployment/order-service
    PartWhat it means
    kubectl rollout undogo back to the previous version. The old ReplicaSet is scaled up again; nothing is rebuilt
    You should see
    deployment.apps/order-service rolled back

    curl INGRESS_ADDRESS/ shows "version":"v1" again within a minute.

  14. Release a broken version

    Terminal
    kubectl set image deployment/order-service app=111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v9-broken
    PartWhat it means
    v9-brokena tag that doesn’t exist
    Terminal
    kubectl get pods
    PartWhat it means
    kubectl get podslist the pods, old and new
    You should see

    The two old pods still 1/1 Running, and one new pod with the status ErrImagePull or ImagePullBackOff, 0/1. Because maxUnavailable is 0, Kubernetes won’t remove an old pod until a new one is ready, so the site keeps working.

    This is where ECS and Kubernetes differ. ECS’s circuit breaker would roll back by itself. Kubernetes stops and waits. kubectl rollout status would wait up to the Deployment’s progress deadline, 10 minutes by default, then report exceeded its progress deadline, and still not roll back. You undo it:

    Terminal
    kubectl rollout undo deployment/order-service
    PartWhat it means
    kubectl rollout undogo back to the previous version, the v1 ReplicaSet, which never stopped running
    deployment/order-servicewhich Deployment
    You should see

    deployment.apps/order-service rolled back, and within a few seconds kubectl get pods shows only healthy pods again.

  15. Clean-up step: Clean up

    Do this in order. The load balancer was built by the controller, not by eksctl. Delete the app while the controller is still running, so it removes the load balancer too.

    1. Delete the app:

    Terminal
    kubectl delete -f order-service.yaml
    PartWhat it means
    kubectl delete -fdelete every object described in the file, including the Ingress, and so the load balancer
    You should see
    deployment.apps "order-service" deleted
    service "order-service" deleted
    ingress.networking.k8s.io "order-service" deleted

    2. Check the load balancer has gone. Wait two minutes, then look in EC2 → Load balancers. There must be no load balancer starting with k8s-default-. If one is still there after five minutes, delete it by hand, and its target group in EC2 → Target groups.

    3. Delete the cluster. This removes the nodes, the node group, the control plane, the service account’s IAM role and the other CloudFormation stacks eksctl made:

    Terminal
    eksctl delete cluster --name quickcart-eks --region us-east-1 --wait
    PartWhat it means
    eksctl delete clusterdelete the cluster and everything eksctl created for it
    --name quickcart-ekswhich cluster
    --region us-east-1where it is
    --waitdon’t return until it’s completely gone
    You should see

    10 to 15 minutes of output, ending in a line saying all cluster resources were deleted.

    4. Delete the policy. IAM → Policies, search for QuickCartLBControllerPolicy, select it, Delete, and type its name to confirm.

    5. Remove the subnet tags (optional, harmless to keep): VPC → Subnets, each of the two subnets, Tags → Manage tags, remove kubernetes.io/role/elb.

    6. The image repository, if you’ve finished with containers: Amazon ECR → Repositories, tick quickcart-orders, Delete, type delete.

    7. Check nothing is left:

    WhereShould show
    EKS → Clustersnothing
    EC2 → Instancesno running instances from this lab; the two nodes show Terminated
    EC2 → Load balancersnothing starting with k8s-
    CloudFormation → Stacksno stacks starting with eksctl-quickcart-eks
    IAM → Identity providersno provider mentioning this cluster. If one is left, delete it
    You should see

    All five checks pass. Nothing from this lab is still billing. Check the Billing page tomorrow to be sure.

Part 5: Check yourself

You delete one of the two pods in a Deployment with replicas set to 2. What happens?

A pod's readiness probe starts failing. What does Kubernetes do?

What does an EKS cluster cost before any pod runs, on standard support?

Why does QuickCart delete its Kubernetes app with kubectl delete before deleting the cluster?

On Kubernetes, why does the order service's pod wait 15 seconds in a preStop hook before stopping?

Part 6: Cheat sheet

On one card
Command or ideaWhat it does
Desired stateYou write what you want; controllers keep making it true
Reconcile loopObserve, compare, act. Over and over
Pod / Deployment / ReplicaSetOne copy / “run N copies” / one version’s set of copies
Service / IngressA stable name inside the cluster / a load balancer from outside
Readiness vs liveness probeTake out of traffic / restart the container
kubectl apply -f FILEMake the cluster match a file
kubectl get podsList pods. describe pod NAME explains problems in its Events
kubectl delete pod NAMEDelete one pod. The Deployment replaces it
kubectl set image deployment/NAME app=IMAGERelease a new image
kubectl rollout status / undoWatch a release / go back one version
EKS control planeAbout $0.10 an hour on standard support. Upgrade before extended support
t3.smallAbout 11 pods, however much CPU is free
Zero downtime on EKSReadiness gate label, preStop sleep, terminationGracePeriodSeconds
Clean upkubectl delete -f first, then eksctl delete cluster

Finished the build and the quiz? Mark it done to fill in its stop on the route map.