Terraform 1.0 is a promise about the next five years

Terraform reached 1.0 in June with essentially no new features, which is the point — the release is a commitment that configuration written today will keep working through the 1.x line. For a tool that had spent six years making breaking changes at every minor version, that is the thing that makes adoption defensible.

The symptom

$ ssh ops@prod-01 'nginx -v; php -v | head -1'
nginx/1.18.0
PHP 7.4.21

$ ssh ops@staging-01 'nginx -v; php -v | head -1'
nginx/1.20.1
PHP 8.0.7

# three environments, built by hand at different times,
# and no record anywhere of what was done to any of them.

$ ls docs/runbooks/
provisioning-2019.md      # last edited March 2019

A runbook from 2019 describing a process that has been performed by hand four times since, each time slightly differently. Rebuilding production from it was not a plausible exercise and nobody had tried.

Why it happens

Infrastructure changes are made where they are needed, under time pressure, and the record is a shell history that is gone by the next login. There is no equivalent of a diff and no equivalent of a review.

The fix

State, and where it must not live

terraform {
  required_version = "~> 1.0"

  backend "s3" {
    bucket         = "tfstate-example"
    key            = "production/terraform.tfstate"
    region         = "eu-west-2"
    encrypt        = true
    dynamodb_table = "tfstate-locks"     # ← the lock
  }
}

The state file holds every value in plain text, including anything marked sensitive, so it must live somewhere with access control and encryption at rest. The lock table is a separate resource that is easy to omit, and without it the backend works perfectly until two people apply at once.

force-unlock exists for a process that died holding the lock and is the most dangerous command in the tool — using it while another apply is genuinely running is how a corrupted state happens. The error names who holds the lock, which is usually enough to resolve it with a message instead.

Importing what already exists, which is the job

$ terraform import aws_s3_bucket.uploads example-uploads
Import successful!

$ terraform plan
  # aws_s3_bucket.uploads must be replaced
  - versioning { enabled = true }
  ~ acl = "private" -> "public-read"

# the configuration must be written to MATCH reality, by
# hand, until the plan is empty. that is the entire job.

Import brings a resource into state and writes no configuration for it, so the loop is import, plan, adjust the configuration until the plan is empty, and move to the next resource. Doing it one resource at a time with an empty plan as the acceptance criterion is the only version that is safe.

Importing twenty things and then writing configuration produces a plan that wants to destroy something, and working out which line caused it is an afternoon. On a production account that afternoon is spent nervously.

# the discipline that made it survivable: one resource,
# one commit, one empty plan
$ terraform plan -detailed-exitcode
$ echo $?
0        # 0 = no changes, 1 = error, 2 = changes pending

# 41 resources imported over three weeks, in 41 commits

Modules that encode a decision

# not a module: eleven pass-through variables
module "database" {
  source            = "./modules/rds"
  engine_version    = "8.0.23"
  instance_class    = "db.t3.medium"
  allocated_storage = 100
  backup_window     = "03:00-04:00"
  # ... seven more
}

# a module: the decisions are inside it
module "database" {
  source = "./modules/rds"
  name   = "shop"
  size   = "medium"    # → instance class, storage, backups,
}                      #   parameter group, monitoring

The test is whether the module encodes a decision. One that says a medium database means this instance class, this backup window and these parameter group settings is worth having, because those decisions are now made once and reviewed once.

The pressure to add the twelfth variable is constant and each addition is individually reasonable. Resisting it means occasionally telling somebody that their case needs a new size rather than a new parameter, which is a better conversation than the one where every caller configures a backup window.

Plan in CI, apply behind a review

jobs:
  plan:
    steps:
      - run: terraform init -input=false
      - run: terraform fmt -check -recursive
      - run: terraform validate
      - run: terraform plan -out=tfplan -input=false
      - run: terraform show -no-color tfplan > plan.txt
      - uses: actions/upload-artifact@v2
        with: { name: plan, path: tfplan }

  apply:
    needs: plan
    environment: production        # ← required reviewers
    steps:
      - run: terraform apply -input=false tfplan

Applying the saved plan rather than re-planning is what guarantees that what was reviewed is what runs — a fresh plan at apply time can differ if anything changed in between, which defeats the review entirely.

The plan output contains the values it intends to write, so posting it to a pull request posts them to anybody who can read the repository. Marking variables sensitive suppresses them in the output and does nothing about the state file, which is a separate problem with a separate answer.

variable "database_password" {
  type      = string
  sensitive = true
}

# in the plan:  + database_password = (sensitive value)
# in the state: the plaintext value, always.

Verifying it worked

$ terraform plan -detailed-exitcode
No changes. Your infrastructure matches the configuration.
$ echo $?
0

$ terraform workspace list
  default
* production
  staging

# and the one that proves the point:
$ cd environments/staging && terraform apply -auto-approve
# a staging environment rebuilt from nothing in 14 minutes,
# matching production's description exactly

An empty plan is the only assertion that matters and it is the acceptance test for the whole exercise. Rebuilding staging from scratch is the second one, and it is what turns “we have Terraform” into “we can rebuild this”.

The drift that Terraform now makes visible is the unexpected benefit and the unexpected annoyance: a change somebody made in a console shows up as a plan diff, which is exactly right and means somebody has to decide whether to codify it or revert it. Four appeared in the first month.

What this costs

A new thing that can be wrong, in a way that takes down infrastructure rather than an application. A plan that says “must be replaced” against a database is a moment requiring full attention, and the tool will happily do it if approved — the review is the only safeguard and it is a human one.

The state file is now a critical artefact with its own backup, access and corruption concerns, and losing it means every resource is orphaned. That is a new single point of failure introduced by a tool adopted for reliability, and versioning on the storage bucket is the mitigation everybody configures after the first scare rather than before.