Kubernetes and EKS
Pods, deployments, and the load balancer controller.
Part 1: The problem
The team can’t agree about Kubernetes. Kofi says it’s the standard; Priya says ECS is enough. Nobody has actually run QuickCart on it, so the argument goes in circles.
So Kofi builds a small EKS cluster one afternoon, runs the order service on it, and shows the team. The demo goes well. Everyone moves on.
Two months later, Ana, reading the AWS bill, finds a cluster nobody remembers, two servers nobody uses, and a load balancer pointing at nothing. The demo was still running.
QuickCart needs two things from this lesson: to understand what Kubernetes really is, so the argument can be settled with facts, and a way to try EKS that is built and torn down completely in one sitting.
Part 2: The idea
Kubernetes does the same job as ECS: it runs your containers, restarts them when they fail, keeps a set number running, and replaces them one by one when you release. It’s open source and runs anywhere: on every major cloud, and on your own servers. EKS is AWS’s managed Kubernetes.
The big idea is how you talk to it. You don’t tell Kubernetes what to do. You tell it what you want, in a file, and it keeps making that true.
The reconcile loop
Inside Kubernetes, small programs called controllers each watch one kind of object. Each one runs the same loop, over and over, for ever:
| Step | What happens | Example |
|---|---|---|
| Observe | Look at what’s really running | 1 pod with the label app: order-service |
| Compare | Compare it with the desired state you wrote | The file says 2. One is missing |
| Act | Change reality to close the gap | Create one pod |
That loop is where self-healing comes from. A pod crashes, a node disappears, someone deletes a pod by mistake: the numbers stop matching, and the controller acts. Nobody has to be awake.
The objects, compared with ECS
| Kubernetes | What it is | Closest thing on ECS |
|---|---|---|
| Pod | One running copy of your app: one or more containers that share a network address | Task |
| Deployment | Says “run N pods of this image” and handles releases | Service |
| ReplicaSet | Created by a Deployment, one per version. Keeps exactly N pods of that version | — ECS does this inside the service |
| Service | A fixed name and address in front of pods that come and go | The target group, roughly |
| Ingress | A request for a load balancer, with routing rules | Load balancer and listener |
| Node | A server that pods run on | Hidden on Fargate |
| Namespace | A named folder for objects, like default or kube-system | The cluster, roughly |
A Deployment doesn’t create pods itself. It creates a ReplicaSet for each version of your app, and the ReplicaSet creates the pods. During a release, the Deployment grows the new ReplicaSet and shrinks the old one. That’s also how kubectl rollout undo is instant: the old ReplicaSet is still there, just scaled to zero.
Probes: is it alive, and is it ready?
Kubernetes checks each container with probes, small requests it makes on a schedule.
| Probe | The question | If it fails |
|---|---|---|
| Readiness probe | Can this pod take traffic right now? | The pod is taken out of the Service. It keeps running, and gets traffic again when the probe passes |
| Liveness probe | Is this container stuck beyond repair? | The container is restarted |
| Startup probe (optional) | Has a slow app finished starting? | The other probes wait until it passes |
This is a real difference from ECS. On ECS, a task failing its load balancer health check is replaced. On Kubernetes, a pod failing its readiness probe is only taken out of traffic, and a failing liveness probe restarts the container in the same pod.
What EKS runs, and what you run
A Kubernetes cluster has two halves.
| Half | What it does | On EKS, who runs it |
|---|---|---|
| Control plane | The API you talk to, the controllers, the scheduler, and the database of desired state | AWS, across several zones. About $0.10 an hour per cluster on standard support, roughly $73 a month |
| Data plane | The nodes, where pods actually run | You, in one of the ways below |
Each Kubernetes version gets about 14 months of standard support on EKS. After that it moves to extended support, which costs about six times more for the control plane, until you upgrade. So an EKS cluster isn’t set up once: it needs upgrading, with its nodes and add-ons, for as long as it exists. Prices change; check the EKS pricing page.
| Where pods run | What it means |
|---|---|
| Managed node group | EC2 instances that AWS launches and helps you patch and replace. You pay for the instances. This lesson uses one |
| Fargate | No nodes to see: each pod gets its own small Fargate machine. Some features, like daemon sets, don’t work |
| Auto Mode | AWS also runs the nodes and core add-ons for you, for an extra fee on top of the instances |
Every pod gets its own IP address from your subnet, and each node size has a limit. A t3.small holds about 11 pods, including Kubernetes’ own system pods, however much CPU is free.
Getting traffic in
A Service gives pods a stable name inside the cluster, but nothing outside can reach it. To get an AWS load balancer, you install the AWS Load Balancer Controller into the cluster. It watches for Ingress objects and builds a real Application Load Balancer for each one.
To build load balancers, the controller needs AWS permissions. You give them to that controller alone, not to every pod on the node, with IAM roles for service accounts (IRSA): a Kubernetes identity linked to an IAM role. It’s the same idea as ECS’s task role. EKS also has a newer way to do this, called Pod Identity; the idea is the same.
Releasing without dropped requests
The settings from Releases without downtime have Kubernetes equivalents:
| ECS | Kubernetes | QuickCart uses |
|---|---|---|
| Min running tasks 100% | maxUnavailable | 0: never go below 2 ready pods |
| Max running tasks 200% | maxSurge | 1: one extra pod at a time |
| Health check | Readiness probe | /health |
| Circuit breaker with rollback | No automatic rollback. The rollout stops and waits | kubectl rollout undo |
Kubernetes needs two extra settings that ECS handles for you. When a pod is removed, Kubernetes takes it out of the Service and sends it SIGTERM at the same moment. The load balancer takes a few seconds to catch up, and without help, requests arrive at a pod that has already gone.
| Setting | What it fixes |
|---|---|
| Pod readiness gate | A new pod only counts as ready once the load balancer says it’s healthy, not just its own probe. So old pods aren’t removed too early |
preStop sleep of 15 seconds | The pod keeps serving for 15 seconds after it’s told to stop, while the load balancer stops sending it traffic. Then SIGTERM arrives, and gunicorn finishes and exits |
terminationGracePeriodSeconds is Kubernetes’ stop timeout. It covers the preStop wait and the app’s shutdown, so QuickCart sets it to 40: 15 for the sleep, 20 for gunicorn’s graceful timeout, and 5 to spare.
Part 3: See it
Here is the order service’s Deployment asking for 2 pods, on a cluster with two nodes. Delete a pod and watch the loop put it back. Then change the number of replicas, and stop node B.
replicas: 2- Observecount the pods
- Comparewith replicas
- Actcreate or delete
- order-service-7d4b9c8f6-x2kqpRunning, ready
- order-service-7d4b9c8f6-m8zrtRunning, ready
What to notice: you never told Kubernetes to create anything. Every pod it made came from the same three steps: observe, compare, act. And the replacement always gets a new name. Pods are disposable: nothing should depend on one particular pod still existing.
Part 4: Do it
Costs money — about $4 a day while it exists: the EKS control plane (about $2.40 a day), two t3.small nodes (about $1 a day together), and a load balancer (about $0.55 a day). Prices are for us-east-1 and change. Delete everything at the end.
You’ll create an EKS cluster, install the load balancer controller, run QuickCart’s order service on it, watch Kubernetes replace a deleted pod, release v2 under load and roll it back, and then delete everything.
You need the quickcart-orders repository in ECR with the v1 and v2 images from Releases without downtime, your load.py, and a working AWS login in your terminal as an IAM user or role, not root. Set up your computer covers the login.
Why eksctl, not the console
You can create an EKS cluster in the console, but it takes two IAM roles you create first, a cluster form, a wait, a separate node group form, and then the terminal anyway, because kubectl and Helm only run there. eksctl is AWS’s recommended command-line tool for EKS. One command builds the control plane, a managed node group, their IAM roles and the networking, and connects kubectl to the new cluster. It uses CloudFormation underneath, so one more command deletes all of it again. This lab uses eksctl, and still uses the console wherever it has an equivalent.
Install the tools
Terminalbrew install eksctl kubectl helmPart What it means brew installinstall programs on your computer with Homebrew eksctlcreates and deletes EKS clusters kubectltalks to any Kubernetes cluster: creates, lists and deletes objects helminstalls packaged add-ons, called charts, into a cluster Check that each one works:
Terminalkubectl version --clientPart What it means kubectl versionprint the version --clientonly this program’s version. There’s no cluster to ask yet You should seeA line starting
Client Version: v1., followed by a version number.eksctl versionandhelm versionprint their versions too. Yours will be newer than anything written here.Check who you're logged in as
The identity that creates the cluster is the one that can use it straight away. Make sure it’s the one you’ll keep using:
Terminalaws sts get-caller-identityPart What it means aws sts get-caller-identityask AWS who this terminal is logged in as You should see{ "UserId": "AIDAEXAMPLEUSERID", "Account": "111122223333", "Arn": "arn:aws:iam::111122223333:user/quickcart-admin" }Your IDs and name will differ. The
Arnmust not end in:root.Find and tag two subnets
In the AWS console: VPCSubnets
Find the default VPC’s subnets in us-east-1a and us-east-1b: their VPC column shows the default VPC. Copy both Subnet IDs, and the VPC ID. They look like
subnet-andvpc-followed by letters and numbers.Then tag each of the two subnets, so the load balancer controller knows it may put an internet-facing load balancer there:
In the AWS console: VPCSubnetsyour us-east-1a subnetTagsManage tagsAdd new tag
Field Enter Key kubernetes.io/role/elbValue 1Choose Save, then do the same for the us-east-1b subnet.
Create the cluster
Put your two subnet IDs in place of
SUBNET_AandSUBNET_B, with a comma and no space between them. The\at the end of a line means the command carries on to the next line.Terminaleksctl create cluster \ --name quickcart-eks \ --region us-east-1 \ --vpc-public-subnets SUBNET_A,SUBNET_B \ --nodegroup-name quickcart-nodes \ --node-type t3.small \ --nodes 2 \ --managedPart What it means eksctl create clusterbuild a new EKS cluster, with everything it needs --name quickcart-eksthe cluster’s name --region us-east-1the region to build it in --vpc-public-subnets SUBNET_A,SUBNET_Buse these two existing subnets in your default VPC, instead of building a new VPC --nodegroup-name quickcart-nodesthe name of the group of servers --node-type t3.smallthe size of each server: 2 vCPUs, 2 GB of memory, about 11 pods --nodes 2two servers, so there’s one in each zone --manageda managed node group: AWS launches, and helps you update, the servers There’s no
--version, on purpose: eksctl picks a current version on standard support.You should seeMany lines over about 15 to 20 minutes, as eksctl builds the control plane and then the nodes. It ends with a line like
EKS cluster "quickcart-eks" in "us-east-1" region is ready. It has also saved the cluster’s details forkubectl, so the next command just works.Check the nodes:
Terminalkubectl get nodesPart What it means kubectl get nodeslist the servers in the cluster You should seeNAME STATUS ROLES AGE VERSION ip-172-31-17-201.ec2.internal Ready <none> 3m v1.33.4-eks-a1b2c3d ip-172-31-90-12.ec2.internal Ready <none> 3m v1.33.4-eks-a1b2c3dYour names, ages and versions will differ. Both must say Ready. In EC2 → Instances there are now two new instances: they’re yours, and on your bill.
Let one pod have its own IAM role
The load balancer controller needs AWS permissions. This command switches on the feature that lets a single Kubernetes identity use an IAM role:
Terminaleksctl utils associate-iam-oidc-provider --cluster quickcart-eks --region us-east-1 --approvePart What it means eksctl utils associate-iam-oidc-providerregister the cluster with IAM as an identity provider, so IAM can trust identities from inside it --cluster quickcart-ekswhich cluster --region us-east-1where the cluster is --approveactually do it. Without this, eksctl only shows what it would do You should seeA line saying the IAM OIDC provider was created for the cluster. In IAM → Identity providers, a new provider whose address mentions
oidc.eks.us-east-1.Create the controller's IAM policy
The controller’s permissions are published as a JSON file called
iam_policy.json, linked from the AWS Load Balancer Controller installation guide in the Amazon EKS documentation. It’s long, and AWS updates it with each controller release, so always take the current file from the guide rather than an old copy. Download it, and open it in a text editor.In the AWS console: IAMPoliciesCreate policyJSON
Delete everything in the editor, then paste in the whole contents of
iam_policy.json. Choose Next.Field Enter Policy name QuickCartLBControllerPolicyChoose Create policy.
You should seeThe policy in the list. Open it and copy its ARN:
arn:aws:iam::111122223333:policy/QuickCartLBControllerPolicy, with your account ID.Give the controller that policy
This step and the next have no console equivalent. Linking an IAM role to an identity inside Kubernetes takes a role with a special trust policy plus a Kubernetes object; eksctl does both in one command. Put your account ID in the ARN:
Terminaleksctl create iamserviceaccount \ --cluster quickcart-eks \ --region us-east-1 \ --namespace kube-system \ --name aws-load-balancer-controller \ --role-name QuickCartLBControllerRole \ --attach-policy-arn arn:aws:iam::111122223333:policy/QuickCartLBControllerPolicy \ --approvePart What it means eksctl create iamserviceaccountcreate a Kubernetes service account (an identity for pods) and an IAM role linked to it --namespace kube-systemput it with Kubernetes’ own system programs --name aws-load-balancer-controllerthe service account’s name. The controller will run as this identity --role-name QuickCartLBControllerRolethe IAM role’s name, so you can find it later --attach-policy-arngive the role the policy you just created --approveactually do it You should seeAbout a minute of output, including a CloudFormation stack being created, and a line saying the service account was created.
Install the controller with Helm
First tell Helm where AWS publishes its EKS charts. The repository address is in the same installation guide, in its Helm section; put it in place of
CHART_REPO_URL:Terminalhelm repo add eks CHART_REPO_URLPart What it means helm repo addremember a chart repository, so you can install from it eksthe short name you’ll use for it CHART_REPO_URLreplace this with the eks-charts repository address from the installation guide You should see"eks" has been added to your repositoriesNow install the controller. Put your VPC ID in place of
VPC_ID:Terminalhelm install aws-load-balancer-controller eks/aws-load-balancer-controller \ --namespace kube-system \ --set clusterName=quickcart-eks \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller \ --set region=us-east-1 \ --set vpcId=VPC_IDPart What it means helm install aws-load-balancer-controllerinstall a chart, and call this installation aws-load-balancer-controllereks/aws-load-balancer-controllerthe chart: from the eksrepository, the one calledaws-load-balancer-controller--namespace kube-systeminstall it next to the service account you made --set clusterName=quickcart-ekstell it which cluster it’s in, so it can name and tag what it builds --set serviceAccount.create=falsedon’t make a new identity --set serviceAccount.name=aws-load-balancer-controlleruse the one eksctl made, with the IAM role --set region=us-east-1and--set vpcId=VPC_IDtell it where it is. Without these, it tries to look them up from the node, which can fail You should seeA few lines ending in
STATUS: deployed, and a note that the controller is installed.Check it’s running:
Terminalkubectl get deployment --namespace kube-system aws-load-balancer-controllerPart What it means kubectl get deploymentshow a Deployment and how many of its pods are ready --namespace kube-systemlook in the kube-systemnamespace, notdefaultaws-load-balancer-controllerwhich Deployment You should seeNAME READY UP-TO-DATE AVAILABLE AGE aws-load-balancer-controller 2/2 2 2 70sIf it shows
0/2, wait a minute and run it again. Count what you just did to connect a load balancer: an identity provider, a policy, a service account, a chart. On ECS it was one section of the service form. That difference belongs in the ECS or EKS decision.Turn on readiness gates
Terminalkubectl label namespace default elbv2.k8s.aws/pod-readiness-gate-inject=enabledPart What it means kubectl labeladd a label to an object namespace defaultthe object: the defaultnamespace, where your app will runelbv2.k8s.aws/pod-readiness-gate-inject=enabledthe label the controller looks for. Pods created in this namespace from now on get a readiness gate You should seenamespace/default labeledFrom now on, a new pod here only counts as ready once the load balancer reports it healthy. Label the namespace before creating the pods: pods that already exist don’t get the gate.
Describe the app
In your
order-servicefolder, createorder-service.yaml. It describes three objects, separated by---lines. Put your account ID in the image address.order-service.yamlapiVersion: apps/v1 kind: Deployment metadata: name: order-service labels: app: order-service spec: replicas: 2 selector: matchLabels: app: order-service strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 0 maxSurge: 1 template: metadata: labels: app: order-service spec: terminationGracePeriodSeconds: 40 securityContext: runAsNonRoot: true runAsUser: 1001 containers: - name: app image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v1 ports: - containerPort: 8080 readinessProbe: httpGet: path: /health port: 8080 periodSeconds: 5 failureThreshold: 2 livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 15 periodSeconds: 10 failureThreshold: 3 resources: requests: cpu: 100m memory: 128Mi limits: memory: 256Mi securityContext: allowPrivilegeEscalation: false lifecycle: preStop: exec: command: ["sleep", "15"] --- apiVersion: v1 kind: Service metadata: name: order-service spec: selector: app: order-service ports: - port: 80 targetPort: 8080 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: order-service annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/healthcheck-path: /health alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30 spec: ingressClassName: alb rules: - http: paths: - path: / pathType: Prefix backend: service: name: order-service port: number: 80The Deployment
Setting What it does If you leave it out When you’d change it apiVersion: apps/v1,kind: DeploymentWhich kind of object this is Required Never metadata.nameThe Deployment’s name, used by kubectlcommandsRequired Never, once created replicas: 2The desired state: 2 pods 1 pod To scale by hand, or let an autoscaler set it selector.matchLabelsWhich pods belong to this Deployment: those labelled app: order-serviceRequired Never. It can’t be changed after creation strategy.rollingUpdate.maxUnavailable: 0During a release, never have fewer than 2 ready pods 25% of the replicas may be missing Higher, to release faster when spare capacity doesn’t matter strategy.rollingUpdate.maxSurge: 1During a release, run at most 1 extra pod 25% extra, rounded up Higher, to release faster when you have spare node capacity template.metadata.labelsLabels on every pod. Must match the selector The Deployment is rejected Add labels for version or team terminationGracePeriodSeconds: 40Kubernetes’ stop timeout: 15 s preStop + 20 s gunicorn + 5 spare 30 seconds: gunicorn could be killed mid-request If your requests or shutdown take longer securityContext.runAsNonRoot: trueRefuse to start the container as root Nothing stops a root container Never securityContext.runAsUser: 1001Run as user 1001, the appuserfrom the DockerfileUses the image’s USEROnly if the image’s user changes containers[].name: appThe container’s name, used by kubectl set imageRequired Never imageThe image to run, from ECR, by tag Required Every release ports.containerPort: 8080The port the app listens on. Mostly documentation, like EXPOSEThe app still works; tools that read it can’t If the app’s port changes readinessProbeAsk /healthevery 5 s; 2 failures in a row takes the pod out of trafficThe pod gets traffic as soon as it starts, before the app is ready Tighter for fast apps, looser for slow ones livenessProbeAsk /healthevery 10 s after 15 s; 3 failures in a row restart the containerA stuck app is never restarted Keep it lenient. See the mistake in the idea resources.requestsWhat the scheduler reserves on a node: a tenth of a CPU and 128 MB The pod may land on a node with no room, and be squeezed Measure real use, then set it a little above resources.limits.memory: 256MiThe most memory the container may use. Above it, it’s killed, shown as OOMKilledIt can use all of a node’s memory, and starve its neighbours When the app really needs more allowPrivilegeEscalation: falseThe process can never gain more privileges than it starts with Allowed Never lifecycle.preStopsleep 15Keep serving for 15 s after being told to stop, while the load balancer catches up Requests arrive at a pod that has already gone Match it to how long your load balancer takes to deregister The Service and the Ingress
Setting What it does If you leave it out When you’d change it Service selector: app: order-serviceSend traffic to pods with this label, as long as they’re ready The Service has no pods behind it Never, while the labels stay the same Service port: 80,targetPort: 8080Inside the cluster, order-service:80reaches the pods on 8080Required If the app’s port changes ingressClassName: albHand this Ingress to the AWS Load Balancer Controller No controller claims it, and no load balancer is built Never, on EKS with this controller alb.ingress.kubernetes.io/scheme: internet-facingA public load balancer internal: reachable only inside the VPCFor services only other services call alb.ingress.kubernetes.io/target-type: ipRegister pod IPs directly, like Fargate tasks instance: traffic goes via a port on each node, an extra hopRarely alb.ingress.kubernetes.io/healthcheck-path: /healthThe ALB’s health check path /, which works here, but isn’t a real readiness checkIf the health path changes target-group-attributes: deregistration_delay.timeout_seconds=30The drain, as on ECS 300 seconds: slow releases Longer than your slowest request ruleswithpath: /andpathType: PrefixSend every path to the order-serviceService on port 80Required To route different paths to different Services This Ingress listens on HTTP port 80 only. For HTTPS you’d add a certificate from Certificate Manager with the
alb.ingress.kubernetes.io/certificate-arnannotation and a listener on 443, as in Domain names and HTTPS.Start it
Terminalkubectl apply -f order-service.yamlPart What it means kubectl applymake the cluster match a file: create what’s new, update what changed -f order-service.yamlthe file to use You should seedeployment.apps/order-service created service/order-service created ingress.networking.k8s.io/order-service createdTerminalkubectl get podsPart What it means kubectl get podslist the pods in the defaultnamespaceYou should seeNAME READY STATUS RESTARTS AGE order-service-7d4b9c8f6-x2kqp 1/1 Running 0 40s order-service-7d4b9c8f6-m8zrt 1/1 Running 0 40sThe part in the middle of each name comes from the ReplicaSet; the end is random. Yours differ.
Now wait for the load balancer:
Terminalkubectl get ingressPart What it means kubectl get ingresslist Ingresses, with the address of the load balancer built for each You should seeNAME CLASS HOSTS ADDRESS PORTS AGE order-service alb * k8s-default-orderser-1a2b3c4d5e-1234567890.us-east-1.elb.amazonaws.com 80 3mThe
ADDRESSappears after two or three minutes; until then, it’s empty. In EC2 → Load balancers, a new load balancer starting withk8s-default-exists. Your YAML file made a real AWS resource.Reach it, and delete a pod
Copy the
ADDRESSfrom the last command:Terminalcurl INGRESS_ADDRESS/Part What it means curlfetch the page and print it INGRESS_ADDRESSreplace this with the address from kubectl get ingressYou should see{"served_by":"order-service-7d4b9c8f6-x2kqp","service":"order-service","version":"v1"}A pod’s hostname is its name, so
served_bytells you which pod answered. Run it a few times: it switches between the two.Now delete one of them, using a name from your
kubectl get pods:Terminalkubectl delete pod POD_NAMEPart What it means kubectl delete poddelete one pod POD_NAMEreplace this with one of your pod names Terminalkubectl get podsPart What it means kubectl get podslist the pods again, to see what Kubernetes did You should seeTwo pods again,
1/1 Running. One has the name you didn’t delete; the other is new, with anAGEof a few seconds. You changed nothing in the file. The reconcile loop saw 1 where you’d asked for 2, and acted.Release v2 under load, then roll back
Window 1: start the load test, with the Ingress address:
Terminalpython3 load.py http://INGRESS_ADDRESS/ 240Part What it means python3 load.pythe load test from Releases without downtime http://INGRESS_ADDRESS/the EKS load balancer this time 240four minutes Window 2: change the image. Put your account ID in:
Terminalkubectl set image deployment/order-service app=111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v2Part What it means kubectl set imagechange the image of a container in an object deployment/order-servicewhich object app=which container: the one named appthe rest the new image: the v2tagYou should seedeployment.apps/order-service image updatedWatch the release:
Terminalkubectl rollout status deployment/order-servicePart What it means kubectl rollout statuswait, printing progress, until the release finishes or fails You should seeLines like
Waiting for deployment "order-service" rollout to finish, one pod at a time because ofmaxSurge: 1, thendeployment "order-service" successfully rolled out. When window 1 finishes, write down its totals. The target isfailed: 0.Now go back to v1, with one command:
Terminalkubectl rollout undo deployment/order-servicePart What it means kubectl rollout undogo back to the previous version. The old ReplicaSet is scaled up again; nothing is rebuilt You should seedeployment.apps/order-service rolled backcurl INGRESS_ADDRESS/shows"version":"v1"again within a minute.Release a broken version
Terminalkubectl set image deployment/order-service app=111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-orders:v9-brokenPart What it means v9-brokena tag that doesn’t exist Terminalkubectl get podsPart What it means kubectl get podslist the pods, old and new You should seeThe two old pods still
1/1 Running, and one new pod with the statusErrImagePullorImagePullBackOff,0/1. BecausemaxUnavailableis 0, Kubernetes won’t remove an old pod until a new one is ready, so the site keeps working.This is where ECS and Kubernetes differ. ECS’s circuit breaker would roll back by itself. Kubernetes stops and waits.
kubectl rollout statuswould wait up to the Deployment’s progress deadline, 10 minutes by default, then reportexceeded its progress deadline, and still not roll back. You undo it:Terminalkubectl rollout undo deployment/order-servicePart What it means kubectl rollout undogo back to the previous version, the v1 ReplicaSet, which never stopped running deployment/order-servicewhich Deployment You should seedeployment.apps/order-service rolled back, and within a few secondskubectl get podsshows only healthy pods again.Clean-up step: Clean up
Do this in order. The load balancer was built by the controller, not by eksctl. Delete the app while the controller is still running, so it removes the load balancer too.
1. Delete the app:
Terminalkubectl delete -f order-service.yamlPart What it means kubectl delete -fdelete every object described in the file, including the Ingress, and so the load balancer You should seedeployment.apps "order-service" deleted service "order-service" deleted ingress.networking.k8s.io "order-service" deleted2. Check the load balancer has gone. Wait two minutes, then look in EC2 → Load balancers. There must be no load balancer starting with
k8s-default-. If one is still there after five minutes, delete it by hand, and its target group in EC2 → Target groups.3. Delete the cluster. This removes the nodes, the node group, the control plane, the service account’s IAM role and the other CloudFormation stacks eksctl made:
Terminaleksctl delete cluster --name quickcart-eks --region us-east-1 --waitPart What it means eksctl delete clusterdelete the cluster and everything eksctl created for it --name quickcart-ekswhich cluster --region us-east-1where it is --waitdon’t return until it’s completely gone You should see10 to 15 minutes of output, ending in a line saying all cluster resources were deleted.
4. Delete the policy. IAM → Policies, search for
QuickCartLBControllerPolicy, select it, Delete, and type its name to confirm.5. Remove the subnet tags (optional, harmless to keep): VPC → Subnets, each of the two subnets, Tags → Manage tags, remove
kubernetes.io/role/elb.6. The image repository, if you’ve finished with containers: Amazon ECR → Repositories, tick
quickcart-orders, Delete, typedelete.7. Check nothing is left:
Where Should show EKS → Clusters nothing EC2 → Instances no running instances from this lab; the two nodes show Terminated EC2 → Load balancers nothing starting with k8s-CloudFormation → Stacks no stacks starting with eksctl-quickcart-eksIAM → Identity providers no provider mentioning this cluster. If one is left, delete it You should seeAll five checks pass. Nothing from this lab is still billing. Check the Billing page tomorrow to be sure.
Part 5: Check yourself
You delete one of the two pods in a Deployment with replicas set to 2. What happens?
A pod's readiness probe starts failing. What does Kubernetes do?
What does an EKS cluster cost before any pod runs, on standard support?
Why does QuickCart delete its Kubernetes app with kubectl delete before deleting the cluster?
On Kubernetes, why does the order service's pod wait 15 seconds in a preStop hook before stopping?
Part 6: Cheat sheet
| Command or idea | What it does |
|---|---|
| Desired state | You write what you want; controllers keep making it true |
| Reconcile loop | Observe, compare, act. Over and over |
| Pod / Deployment / ReplicaSet | One copy / “run N copies” / one version’s set of copies |
| Service / Ingress | A stable name inside the cluster / a load balancer from outside |
| Readiness vs liveness probe | Take out of traffic / restart the container |
kubectl apply -f FILE | Make the cluster match a file |
kubectl get pods | List pods. describe pod NAME explains problems in its Events |
kubectl delete pod NAME | Delete one pod. The Deployment replaces it |
kubectl set image deployment/NAME app=IMAGE | Release a new image |
kubectl rollout status / undo | Watch a release / go back one version |
| EKS control plane | About $0.10 an hour on standard support. Upgrade before extended support |
| t3.small | About 11 pods, however much CPU is free |
| Zero downtime on EKS | Readiness gate label, preStop sleep, terminationGracePeriodSeconds |
| Clean up | kubectl delete -f first, then eksctl delete cluster |