The state file two people wrote to at once

Two pipelines running against one state file, and a plan that proposed destroying a database because it had been applied from a state the other pipeline had already moved. The lock existed and did not help, because the run that held it had been cancelled and the lock outlived it.

The symptom

$ terraform plan
Error: Error acquiring the state lock
  ID:        6f0a8c2e-1d4b-7a91-b3c5-0e8f2a4d9b17
  Path:      infra/terraform.tfstate
  Operation: OperationTypeApply
  Who:       runner@fv-az412-104
  Created:   2023-07-19 08:41:02 UTC

$ date -u
2023-07-19 09:22:11 UTC

# 41 minutes. the run was cancelled at 08:43.

and what happened next, which is the actual incident:

  09:24  somebody ran force-unlock
  09:25  terraform plan
         Plan: 2 to add, 1 to change, 1 to DESTROY
           - aws_db_instance.reporting
  09:26  the plan was not applied, because the word
         DESTROY was in it and somebody read it.

the destroy was proposed because the cancelled apply had
created a replacement instance that was not in state.

The plan was correct given the state it had. A cancelled apply had created a resource, failed to record it, and the next plan saw a configuration describing something the state said did not exist — so it proposed reconciling by destroying what was there.

Why it happens

A lock with no lease is held until something releases it, and a cancelled process releases nothing. The state file is then guarded against concurrent access by a lock nobody owns, and the recovery procedure is a command that removes the guard.

The fix

Releasing the lock on cancellation

- name: release lock on cancellation
  if: cancelled()
  run: |
    id=$(terraform force-unlock -force 2>&1 
         | grep -oP 'ID:s+KS+' || true)
    [ -n "$id" ] && terraform force-unlock -force "$id"
    echo 'lock released after cancellation' >> "$GITHUB_STEP_SUMMARY"

The if: cancelled() step runs when the job is cancelled rather than when it fails, which is the case the lock outlives. This does not make the cancelled apply safe — it makes the next run able to start, which is a different and smaller problem.

Splitting state by lifecycle

one state file for everything meant every plan touched
everything, and a lock on the network blocked a change
to a DNS record.

the split, by how often things change:

  infra/network/     VPC, subnets, security groups.
                     changed 3 times in 2 years.
  infra/data/        databases, caches, buckets.
                     changed monthly.
  infra/app/         compute, DNS, certificates.
                     changed weekly.

and the dependency: app reads network and data outputs
via a remote state data source, one direction only.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "turkerdev-tfstate"
    key    = "network/terraform.tfstate"
  }
}

resource "aws_instance" "app" {
  subnet_id = data.terraform_remote_state.network.outputs.app_subnet_id
}

Reading another state’s outputs rather than importing its resources is what keeps the dependency one-directional. It also means the network state can be locked for an hour without blocking an application deploy, which was the day-to-day irritation that prompted the split.

Applying only a saved plan

  plan:
    steps:
      - run: terraform plan -out=tfplan -lock-timeout=5m
      - run: terraform show -no-color tfplan > plan.txt
      - uses: actions/upload-artifact@v3
        with: { name: tfplan, path: tfplan }
      - run: grep -q 'destroy' plan.txt &&
          echo 'DESTROY_IN_PLAN=true' >> "$GITHUB_ENV" || true

  apply:
    needs: plan
    environment: production      # required reviewer
    steps:
      - uses: actions/download-artifact@v3
        with: { name: tfplan }
      - run: terraform apply -lock-timeout=5m tfplan

Applying a saved plan rather than re-planning is what makes the review meaningful — a plan generated at apply time may differ from the one somebody approved. The destroy detection sets a flag the environment protection can key on, so a plan containing a destruction requires a second reviewer.

The checklist before force-unlock

  1  is the holding run actually dead? check the CI run
     page, not the lock's age — a 15-minute apply is
     normal here.
  2  did the apply partially complete? read the run's
     log; every created resource is printed. write
     them down.
  3  take a state backup, now, by hand.
  4  force-unlock, with the ID from the error.
  5  plan, and read every line. if it proposes creating
     something step 2 says already exists, import it
     rather than applying.

step 2 is the one that was skipped in July.

Reading the cancelled run’s log to find what it created is the step that turns a dangerous recovery into a safe one, and it takes four minutes. Skipping it is how a plan that proposes a destroy gets applied by somebody who assumes the tool knows better.

Recovering the state we broke

$ aws s3api list-object-versions --bucket turkerdev-tfstate 
    --prefix data/terraform.tfstate --max-items 5 
    --query 'Versions[].[VersionId,LastModified]' --output table

$ aws s3api get-object --bucket turkerdev-tfstate 
    --key data/terraform.tfstate --version-id 'Xr8...' 
    ./recovered.tfstate

$ terraform state push ./recovered.tfstate
$ terraform plan
No changes. Your infrastructure matches the configuration.

Bucket versioning is what made this a fifteen-minute recovery rather than a rebuild, and it has to be enabled before it is needed. The state bucket also has a lifecycle rule keeping versions for ninety days, which is the only backup this file has.

Verifying it worked

# two pipelines, started deliberately within 5 seconds
$ gh workflow run infra.yml -f target=app &
$ gh workflow run infra.yml -f target=app &

  run 1: acquired lock, applied, released.   2m 10s
  run 2: waiting for lock... acquired at 2m 12s.
         Plan: no changes.

# and a cancellation drill
$ gh run cancel <id>    # mid-apply
  lock released after cancellation
$ terraform plan -lock-timeout=30s
  # acquired immediately; 1 resource to import

$ aws s3api get-bucket-versioning --bucket turkerdev-tfstate
{ "Status": "Enabled" }

Running two pipelines concurrently on purpose is the test, and the second one waiting rather than failing is the outcome. The cancellation drill leaving one resource to import is the honest result — the lock is released and the state is still behind reality, which is what the checklist exists for.

What this costs

Three state files instead of one, with a dependency order between them that somebody has to know. A change spanning the network and the application is now two applies in sequence, and getting the order wrong produces a plan that references an output that does not exist yet.

The cancellation handler is also a small lie: it releases the lock and does not repair the state, so it makes the next run possible rather than correct. That is deliberate — an automated state repair would be considerably more dangerous than a blocked pipeline — and it means the checklist is still load-bearing after all of this.