Fourteen resources created by hand during an incident in 2019 and referenced in the configuration by hard-coded identifier ever since. The infrastructure code described a system it did not fully manage, and everybody knew it and nobody had a safe way to fix it.
The symptom
# what the configuration looked like
resource "aws_instance" "app" {
subnet_id = "subnet-0a4f8c2e1d7b" # created 2019-08, by hand
vpc_security_group_ids = [
"sg-08c1f4a7e9b2", # ditto
]
}
# 14 of these. every one a value that terraform cannot
# see, cannot verify, and cannot recreate.what that meant: a plan could not detect drift on them,
a rebuild in a new region was impossible, the disaster
recovery document said "recreate the security groups
from the screenshots", and one of them had a rule added
in 2021 that nobody could account for.Why it happens
An incident creates a resource by hand because that is faster, and the intention is always to bring it under management afterwards. The old import path was a command run against state, unreviewable and irreversible, so nobody wanted to be the one to run it.
The fix
An import block, in a plan you can read
import {
to = aws_security_group.app
id = "sg-08c1f4a7e9b2"
}
resource "aws_security_group" "app" {
name = "app-sg"
vpc_id = aws_vpc.main.id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = [aws_security_group.lb.id]
}
}$ terraform plan
Terraform will perform the following actions:
# aws_security_group.app will be imported
resource "aws_security_group" "app" {
id = "sg-08c1f4a7e9b2"
}
# aws_security_group.app will be updated in-place
~ resource "aws_security_group" "app" {
- ingress {
- from_port = 22
- cidr_blocks = ["0.0.0.0/0"]
}
}
Plan: 1 to import, 1 to change.The plan shows the import and the drift in one output, which the old command could not — it imported, and the next plan told you what was wrong, in a separate step nobody connected to the import. The SSH-from-anywhere rule in that diff is the 2021 addition nobody could account for.
Generating configuration, and reading it critically
$ terraform plan -generate-config-out=generated.tf
$ wc -l generated.tf
412
# for 14 resources. it emits every attribute, including
# every default, including the computed ones.
$ grep -c 'arns*=' generated.tf
14 # every one of which must be deleted: computedThe generated file is a first draft and reviewing it down to what is intentional is the actual work — four hundred lines became ninety. Leaving the computed attributes in produces a configuration that fights the provider on every plan, which is a worse state than not managing the resource at all.
The three that had drifted
the SSH rule 0.0.0.0/0 on port 22, added during a
2021 incident → removed; the bastion path had existed
since 2020
a bucket lifecycle the configuration said 90 days, the
bucket said 30. somebody changed it in the console to
reduce a bill → 90 days, and the bill discussed
an instance type t3.medium in code, t3.large in
reality — scaled up during a launch, never recorded
→ t3.large, and the plan stops proposing a shrinkThe instance type is the one that would have caused an outage: bringing it under management without noticing would have produced a plan that resized production down, and applying it during a routine change would have been a surprise capacity reduction.
Check blocks for the runbook assertions
check "app_health" {
data "http" "app" {
url = "https://${var.hostname}/health"
}
assert {
condition = data.http.app.status_code == 200
error_message = "health returned ${data.http.app.status_code}"
}
}
check "no_public_ssh" {
assert {
condition = !contains(
flatten(aws_security_group.app.ingress[*].cidr_blocks),
"0.0.0.0/0"
)
error_message = "a security group allows ingress from anywhere"
}
}The second check encodes the thing we had just found, so it cannot come back silently. A failing check is a warning rather than an error and the exit code stays zero, which means CI has to grep for it — that is a rough edge and it is the price of checks not blocking an otherwise successful apply.
Removing the import blocks afterwards
an import block is idempotent: once the resource is in
state, it is a no-op. so leaving them costs nothing
except clutter.
we removed them after every environment had applied,
which took a fortnight and a checklist, because a block
removed before an environment applies means that
environment tries to CREATE the resource.
the check that made this safe:
terraform state list | grep -c aws_security_group.app
in each environment, before removing anything.Verifying it worked
$ terraform plan
No changes. Your infrastructure matches the configuration.
$ terraform plan # a week later, unchanged
No changes.
$ grep -cE '"(sg|subnet|vpc)-[0-9a-f]+"' *.tf
0 # was 14 hard-coded identifiers
$ terraform plan -var-file=dr.tfvars -state=/dev/null 2>&1 | tail -1
Plan: 41 to add, 0 to change, 0 to destroy.
# the whole stack is now reproducible in a second regionThe last one is the outcome that justified the exercise: a plan against an empty state proposing to create everything means the disaster recovery document can stop referring to screenshots. Two consecutive no-change plans a week apart is the assertion that nothing is fighting the provider.
What this costs
Configuration generated by a tool and owned by us. Ninety lines that nobody wrote from scratch, describing resources whose original intent is lost — the review was careful and it is still a description reverse-engineered from a running system rather than a statement of what was meant.
Bringing resources under management also means a plan can now destroy them. Fourteen resources that were previously immune to a bad plan are now within reach of one, which is the trade: manageable and therefore breakable, against unmanaged and therefore irreproducible.