Operating Terraform

AWS and the code had quietly disagreed for three weeks. That is drift, and dealing with it is part of running Terraform day to day, after the first build.

This lesson covers the tools for living with Terraform: spotting drift, adopting things made by hand, renaming without destroying, and getting out of trouble.

Building with Terraform is the easy part. After that, the job is to keep three things agreeing: the code, the state and AWS. People click in the console. Code gets reorganised. Applies get interrupted. Providers release new versions.

Drift: when AWS and the code disagree

Every terraform plan starts by refreshing: it reads the real settings of everything in the state from AWS. Then it compares them with the code. If someone changed a setting by hand, the plan offers to change it back. The code wins.

So when a plan shows a change you didn’t make, there are two honest answers:

The dishonest answer is to apply without reading, which is how Leo brought the outage back.

A refresh-only plan, made with terraform plan -refresh-only, is a different kind of plan. It shows only what changed outside Terraform, and it never plans to undo anything. It’s useful when you’re investigating. If you apply it, Terraform records the new values in the state and still changes nothing in AWS. It does not change your code, so on its own it fixes nothing. The next normal plan still wants to undo the change.

Catching drift on a schedule

A pipeline can run a plan every night and alert someone if anything differs. It uses -detailed-exitcode, which makes the plan’s exit code mean something:

The pipeline project runs exactly this.

Adopting things made by hand: import

Lots of things are made before anyone writes Terraform: a repository clicked together in a hurry, a bucket from the early days. If you just write a resource block for one, Terraform plans to create it, and AWS refuses because it already exists.

An import block tells Terraform: this resource in the code is that thing which already exists.

import {
  to = aws_ecr_repository.service["payments"]
  id = "quickcart-payments"
}

Import blocks show up in the plan as will be imported, so they’re reviewed like any other change. The import only happens when you apply. Afterwards, delete the block. Leaving it in is harmless, but it’s clutter.

If the plan shows changes as well as the import, the code doesn’t match reality yet. Import doesn’t change the code. It only adopts the object, and after that Terraform makes it match the code. So either the code is right and you apply, or you edit the code until the only thing left is the import.

For a lot of resources at once, terraform plan -generate-config-out=generated.tf writes a first draft of the resource blocks for any import block that has none. Treat that file as a draft: tidy it and move the blocks into your own files.

The older way: the terraform import command

terraform import 'aws_ecr_repository.service["payments"]' quickcart-payments

aws_ecr_repository.service["payments"]: Importing from ID "quickcart-payments"...
aws_ecr_repository.service["payments"]: Import prepared!
  Prepared aws_ecr_repository for import
aws_ecr_repository.service["payments"]: Refreshing state... [id=quickcart-payments]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

It changes the state straight away, with no plan and nothing to review. It still works, and you’ll see it in older guides. Prefer the block.

Renaming without destroying: moved

Terraform identifies each resource by its resource address: the type and the name you gave it in the code, like aws_ecr_repository.orders. Change the address, and as far as Terraform is concerned, one thing was deleted from the code and another was added.

A moved block says that the old address and the new one are the same thing:

moved {
  from = aws_ecr_repository.orders
  to   = aws_ecr_repository.service["orders"]
}

Examples of from and to pairs:

The plan then says has moved to, and nothing in AWS changes. Keep the moved block for a while, until every copy of the state has been applied with it: every environment, and anyone’s branch. For a module that other teams use, keep it for good. It costs nothing.

terraform state mv does the same rename directly in the state, from the command line. Like terraform import, it changes the state at once with no plan and no review. Use a moved block instead.

Letting go without destroying: removed

Sometimes something should stay in AWS but leave this code. For example, another team takes it over. If you just delete the resource block, the next plan destroys it.

A removed block makes Terraform forget it instead:

removed {
  from = aws_ecr_repository.reports

  lifecycle {
    destroy = false
  }
}

The plan says so in plain words:

 # aws_ecr_repository.reports will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)

To forget one instance of a for_each or count, there’s no block. Use the command, which changes the state straight away:

terraform state rm 'aws_ecr_repository.service["payments"]'

Removed aws_ecr_repository.service["payments"]
Successfully removed 1 resource instance(s).

Add -dry-run first to see what it would remove without removing it. Then take the item out of the code, or the next plan will offer to create it again.

Targeted plans, and why to avoid them

-target limits a plan or apply to one resource and whatever it depends on:

terraform plan -target='aws_ecr_repository.service["orders"]'

Terraform warns you every time:

Warning: Resource targeting is in effect

You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.

The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.

Take the warning literally. Everything outside the target goes unplanned, so drift and unapplied changes pile up out of sight. If a team reaches for -target every week, the configuration is too big. Split it into smaller parts with their own state, so each full plan is quick.

Recovering from mistakes

Before any repair, save a copy of the state:

terraform state pull > state-backup.json

Then find your situation:

The pattern in every row: plan first, read it, and never apply a plan you can’t explain.

Upgrading providers

Two things decide which AWS provider you run:

A plain terraform init installs exactly the version in the lock file, so everyone on the team runs the same one. terraform init -upgrade looks for the newest version your range allows and updates the lock file.

Maya reorganises the code: the orders repository moves into a for_each, or into a module. Nothing about the repository itself changes. Plan it without a moved block and then with one, and apply each to see what happens to the repository.

$ terraform plan

  # aws_ecr_repository.orders will be destroyed
  # (because aws_ecr_repository.orders is not in configuration)
  - resource "aws_ecr_repository" "orders" {

  # aws_ecr_repository.service["orders"] will be created
  + resource "aws_ecr_repository" "service" {

Plan: 1 to add, 0 to change, 1 to destroy.

What to notice: the only difference between the two plans is four lines of code that never touch AWS. Without them, Terraform can only see an address that disappeared and a new one that appeared, so a harmless rename turns into a destroy.

Free — two empty ECR repositories. ECR only charges for stored images, and these hold none.

You’ll build a repository with Terraform, drift it by hand, fix the drift, refactor the code with a moved block, adopt a repository made in the console, and upgrade the provider. About 40 minutes.

You need Terraform 1.10 or later and a working AWS login in your terminal. Set up your computer covers both.

1. Build the starting point

   Make a new folder and move into it:

   TerminalCopy

   mkdir quickcart-ops

   Part	What it means
   mkdir	make a directory
   quickcart-ops	its name

   TerminalCopy

   cd quickcart-ops

   Part	What it means
   cd	change directory. Every later command runs in here
   quickcart-ops	the folder you just made

   Create main.tf:

   main.tfCopy

   terraform {
     required_version = ">= 1.10.0"
   
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 6.0"
       }
     }
   }
   
   provider "aws" {
     region = "us-east-1"
   }
   
   resource "aws_ecr_repository" "orders" {
     name                 = "quickcart-orders"
     image_tag_mutability = "IMMUTABLE"
     force_delete         = true
   
     tags = {
       Project = "quickcart"
     }
   }

   Setting	What it does	If you leave it out	When you’d change it
   required_version	Refuse to run on a Terraform older than 1.10	An older Terraform might half-understand the files	When you rely on a newer feature
   required_providers → aws	Use the hashicorp/aws provider, any 6.x version	Terraform picks the newest version, even a new major one	For a deliberate major upgrade
   provider "aws" → region	Build in us-east-1	Terraform uses the region from your AWS login settings	To build somewhere else
   aws_ecr_repository "orders"	A private store for container images. Its address in the code is aws_ecr_repository.orders	—	—
   name	The repository’s name in AWS	Required	Never. Changing it replaces the repository
   image_tag_mutability	IMMUTABLE: a tag like v1 can never be moved to a different image	Defaults to MUTABLE	Rarely. See Storing images: ECR
   force_delete	Let terraform destroy delete the repository even if it holds images	Destroy fails while any image is inside	Set it to false for real repositories
   tags	Labels for finding and costing things	No labels	Add owner and environment in real projects

   TerminalCopy

   terraform init

   Part	What it means
   terraform init	download the AWS provider and prepare the folder

   You should see

   Terraform has been successfully initialized!

   TerminalCopy

   terraform apply

   Part	What it means
   terraform apply	plan, show the plan, and carry it out once you type yes

   Type yes when asked.

   You should see

   aws_ecr_repository.orders: Creating...
   aws_ecr_repository.orders: Creation complete after 1s [id=quickcart-orders]
   
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

2. Drift it by hand, and catch it

   Play Noor at 2am. Change the repository in the console, not in the code:

   AWSIn the AWS console: ECR→Private registry→Repositories→tick quickcart-orders→Actions→Edit

   Field	Enter
   Image tag mutability	Mutable

   Choose Save. Now ask Terraform what it thinks:

   TerminalCopy

   terraform plan

   Part	What it means
   terraform plan	read the real repository from AWS, compare it with the code, and show the difference

   You should see

   Terraform will perform the following actions:
   
     # aws_ecr_repository.orders will be updated in-place
     ~ resource "aws_ecr_repository" "orders" {
           id                   = "quickcart-orders"
         ~ image_tag_mutability = "MUTABLE" -> "IMMUTABLE"
           name                 = "quickcart-orders"
           tags                 = {
               "Project" = "quickcart"
           }
           # (6 unchanged attributes hidden)
   
           # (2 unchanged blocks hidden)
       }
   
   Plan: 0 to add, 1 to change, 0 to destroy.

   Read it: the value in AWS is MUTABLE, and the code wants IMMUTABLE. Nobody changed the code, so this line is drift.

   Now look at the same drift the other way round:

   TerminalCopy

   terraform plan -refresh-only

   Part	What it means
   terraform plan	make a plan
   -refresh-only	only show what changed outside Terraform. Plan nothing to undo it

   You should see

   Note: Objects have changed outside of Terraform
   
   Terraform detected the following changes made outside of Terraform since the
   last "terraform apply" which may have affected this plan:
   
     # aws_ecr_repository.orders has changed
     ~ resource "aws_ecr_repository" "orders" {
           id                   = "quickcart-orders"
         ~ image_tag_mutability = "IMMUTABLE" -> "MUTABLE"
           name                 = "quickcart-orders"
           tags                 = {
               "Project" = "quickcart"
           }
           # (6 unchanged attributes hidden)
   
           # (2 unchanged blocks hidden)
       }
   
   
   This is a refresh-only plan, so Terraform will not take any actions to undo
   these. If you were expecting these changes then you can apply this plan to
   record the updated values in the Terraform state without changing any remote
   objects.

   The arrow now points the other way: from what the state remembers to what AWS has.

   This time the hand-made change was wrong, since immutable tags are what QuickCart wants. So put it back:

   TerminalCopy

   terraform apply

   Part	What it means
   terraform apply	carry out the plan: set the repository back to IMMUTABLE

   You should see

   Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

   Finally, the check a nightly pipeline runs:

   TerminalCopy

   terraform plan -detailed-exitcode

   Part	What it means
   terraform plan	make a plan
   -detailed-exitcode	finish with exit code 0 for no changes, 1 for an error, 2 for changes

   TerminalCopy

   echo $?

   Part	What it means
   echo	print something
   $?	your shell’s record of the last command’s exit code

   You should see

   0

   0: no drift. If you run the pair again after another console change, you’ll see 2.

3. Refactor into for_each, safely

   QuickCart is about to have more than one service, so Maya wants one resource block that makes a repository per service. Replace main.tf with this version. Only the resource block changes:

   main.tfCopy

   terraform {
     required_version = ">= 1.10.0"
   
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 6.0"
       }
     }
   }
   
   provider "aws" {
     region = "us-east-1"
   }
   
   resource "aws_ecr_repository" "service" {
     for_each = toset(["orders"])
   
     name                 = "quickcart-${each.key}"
     image_tag_mutability = "IMMUTABLE"
     force_delete         = true
   
     tags = {
       Project = "quickcart"
     }
   }

   Setting	What it does	If you leave it out	When you’d change it
   "service"	The new name inside Terraform. The address becomes aws_ecr_repository.service["orders"]	—	—
   for_each	One repository per item in the set. toset turns the list into a set, which for_each needs. For now the set holds only orders	One repository only	Add a service name to add a repository. You’ll add payments in the next step
   name	quickcart- followed by the item, so the orders repository keeps the name quickcart-orders	Required	—
   the other settings	Unchanged from step 1	—	—

   Plan:

   TerminalCopy

   terraform plan

   Part	What it means
   terraform plan	compare the reorganised code with the state

   The summary lines say

     # aws_ecr_repository.orders will be destroyed
     # (because aws_ecr_repository.orders is not in configuration)
     # aws_ecr_repository.service["orders"] will be created
   Plan: 1 to add, 0 to change, 1 to destroy.

   That’s the Case from the top of this lesson. Same repository, same name, but a new address. Applying would delete quickcart-orders and every image in it. Don’t apply.

   Create moved.tf next to main.tf:

   moved.tfCopy

   moved {
     from = aws_ecr_repository.orders
     to   = aws_ecr_repository.service["orders"]
   }

   Each setting is explained in the idea, above.

   TerminalCopy

   terraform plan

   Part	What it means
   terraform plan	plan again, now that Terraform knows the two addresses are the same

   You should see

   Terraform will perform the following actions:
   
     # aws_ecr_repository.orders has moved to aws_ecr_repository.service["orders"]
       resource "aws_ecr_repository" "service" {
           id                   = "quickcart-orders"
           name                 = "quickcart-orders"
           tags                 = {
               "Project" = "quickcart"
           }
           # (7 unchanged attributes hidden)
   
           # (2 unchanged blocks hidden)
       }
   
   Plan: 0 to add, 0 to change, 0 to destroy.

   TerminalCopy

   terraform apply

   Part	What it means
   terraform apply	record the new address in the state. Nothing in AWS changes

   You should see

   Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

4. Adopt a repository made in the console

   Someone made the payments repository by hand before QuickCart used Terraform. Make it now, the same way:

   AWSIn the AWS console: ECR→Private registry→Repositories→Create repository

   Field	Enter
   Repository name	quickcart-payments
   Image tag mutability	Mutable, the default. Leave it
   Encryption settings	AES-256, the default. Leave it

   Choose Create.

   Now add payments to the set in main.tf. The whole file now reads:

   main.tfCopy

   terraform {
     required_version = ">= 1.10.0"
   
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 6.0"
       }
     }
   }
   
   provider "aws" {
     region = "us-east-1"
   }
   
   resource "aws_ecr_repository" "service" {
     for_each = toset(["orders", "payments"])
   
     name                 = "quickcart-${each.key}"
     image_tag_mutability = "IMMUTABLE"
     force_delete         = true
   
     tags = {
       Project = "quickcart"
     }
   }

   Then create import.tf next to it:

   import.tfCopy

   import {
     to = aws_ecr_repository.service["payments"]
     id = "quickcart-payments"
   }

   Each setting is explained in the idea, above. For an ECR repository, the id is its name.

   TerminalCopy

   terraform plan

   Part	What it means
   terraform plan	plan the import, and compare the imported repository with the code

   You should see

     # aws_ecr_repository.service["payments"] will be updated in-place
     # (imported from "quickcart-payments")
     ~ resource "aws_ecr_repository" "service" {
           arn                  = "arn:aws:ecr:us-east-1:111122223333:repository/quickcart-payments"
         + force_delete         = true
           id                   = "quickcart-payments"
         ~ image_tag_mutability = "MUTABLE" -> "IMMUTABLE"
           name                 = "quickcart-payments"
           region               = "us-east-1"
           registry_id          = "111122223333"
           repository_url       = "111122223333.dkr.ecr.us-east-1.amazonaws.com/quickcart-payments"
         ~ tags                 = {
             + "Project" = "quickcart"
           }
         ~ tags_all             = {
             + "Project" = "quickcart"
           }
   
           encryption_configuration {
               encryption_type = "AES256"
               kms_key         = null
           }
   
           image_scanning_configuration {
               scan_on_push = false
           }
       }
   
   Plan: 1 to import, 0 to add, 1 to change, 0 to destroy.

   Your account number will be your own, and there may be a few more lines. Read the ~ lines: the repository made by hand is mutable and has no tags. The code wants immutable tags and the Project tag. That’s QuickCart’s standard, so the code is right here. Apply:

   TerminalCopy

   terraform apply

   Part	What it means
   terraform apply	import the repository into the state, then bring it in line with the code

   You should see

   aws_ecr_repository.service["payments"]: Importing... [id=quickcart-payments]
   aws_ecr_repository.service["payments"]: Import complete [id=quickcart-payments]
   aws_ecr_repository.service["payments"]: Modifying... [id=quickcart-payments]
   aws_ecr_repository.service["payments"]: Modifications complete after 0s [id=quickcart-payments]
   
   Apply complete! Resources: 1 imported, 0 added, 1 changed, 0 destroyed.

   Delete import.tf: its job is done. Run terraform plan once more and you should see No changes. Your infrastructure matches the configuration.

5. Check for provider upgrades

   TerminalCopy

   terraform init -upgrade

   Part	What it means
   terraform init	prepare the folder
   -upgrade	look for the newest provider allowed by ~> 6.0, instead of the version in the lock file

   If there’s nothing newer, you’ll see:

   You should see

   - Finding hashicorp/aws versions matching "~> 6.0"...
   - Using previously-installed hashicorp/aws v6.66.0

   If there is, you’ll see it installed, and a note that the lock file changed:

   You should see

   - Finding hashicorp/aws versions matching "~> 6.0"...
   - Installing hashicorp/aws v6.67.0...
   - Installed hashicorp/aws v6.67.0 (signed by HashiCorp)
   
   Terraform has made some changes to the provider dependency selections recorded
   in the .terraform.lock.hcl file. Review those changes and commit them to your
   version control system if they represent changes you intended to make.

   Your version numbers will be different. Either way, plan straight after:

   TerminalCopy

   terraform plan

   Part	What it means
   terraform plan	check the new provider sees everything the same way the old one did

   You should see

   No changes. Your infrastructure matches the configuration.

   A clean plan means the upgrade is safe to commit. Any ~ here that nobody asked for comes from the upgrade, and is a reason to read the provider’s changelog before going further.

6. Clean-up step: Clean up

   Both repositories are now managed by Terraform, including the one you made by hand. So one command removes both:

   TerminalCopy

   terraform destroy

   Part	What it means
   terraform destroy	plan the removal of everything in the state, and carry it out once you type yes

   You should see

   Destroy complete! Resources: 2 destroyed.

   Check ECR → Private registry → Repositories. Neither quickcart-orders nor quickcart-payments should be listed.