OIDC to the cloud, so the pipeline holds no long-lived key

The deploy credential in the pipeline settings had been created in March 2018 by somebody who had left in 2019. It had no expiry, appeared in no audit log, and nobody could say with confidence which of the four people with repository admin could read it. GitHub shipped OIDC support for cloud providers in November, which replaces it with a token that lives for fifteen minutes.

The symptom

$ gh api repos/org/shop/actions/secrets -q '.secrets[] | 
    "(.name)t(.created_at)t(.updated_at)"'
AWS_ACCESS_KEY_ID       2018-03-14  2018-03-14
AWS_SECRET_ACCESS_KEY   2018-03-14  2018-03-14
DEPLOY_SSH_KEY          2019-08-02  2019-08-02

# never rotated. and on the other side:
$ aws iam get-access-key-last-used --access-key-id AKIA...
{ "LastUsedDate": "2021-11-08T09:41:00Z",
  "ServiceName": "s3", "Region": "eu-west-2" }

# used daily, for three and a half years.

A credential that is used daily and has never been rotated is not an oversight — rotating it means coordinating a change in two systems with a window where deploys fail, and there was never a good moment. That is why long-lived keys stay long-lived.

Why it happens

The pipeline needs to deploy and a static key was the only mechanism available, so one was created and pasted into a settings page. Every property that makes it uncomfortable — no expiry, no audit, no scope beyond the IAM policy — follows from it being a shared secret rather than an identity.

The fix

The exchange, and what it replaces

before:
  a static key, in a settings page, forever
  → anybody who can read it can use it, from anywhere

after:
  the runner requests a signed JWT from GitHub, describing
  the repository, branch, workflow and environment
  → the cloud provider validates the signature against
    GitHub's published keys, checks the claims against a
    trust policy, and issues a 15-minute credential

what is stored: nothing. the trust policy is the config,
and it is in Terraform rather than in a settings page.

The credential existing for fifteen minutes and only inside a specific job is the whole benefit. There is nothing to leak from a settings page, nothing to rotate, and every use appears in the cloud provider’s audit log with the workflow that requested it.

Scoping the trust, which is where the value is

{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::...:oidc-provider/token.actions.githubusercontent.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub":
        "repo:org/shop:environment:production"
    }
  }
}

Scoping to an environment rather than a branch is the strongest form available, because it composes with the required-reviewer gate — a job without environment: production cannot obtain the credential at all, no matter what branch it runs from.

the subject claim, from most to least specific:

  repo:org/shop:environment:production
  repo:org/shop:ref:refs/heads/main
  repo:org/shop:ref:refs/tags/v*        (StringLike)
  repo:org/shop:*                        every workflow
  repo:*                                 EVERY REPOSITORY
                                         ON GITHUB

the last one is the misconfiguration, and the policy
looks complete because the audience check still passes.

Omitting the subject condition entirely means any GitHub Actions workflow anywhere can assume the role, which is a total compromise wearing the appearance of a working configuration. Every one of these is a string match, so a typo fails closed — which is the one merciful property of the whole arrangement.

The workflow side

jobs:
  deploy:
    environment: production
    permissions:
      id-token: write        # ← without this, no token is issued
      contents: read

    steps:
      - uses: actions/checkout@v2

      - uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::...:role/github-deploy
          aws-region: eu-west-2

      - run: aws s3 sync ./public s3://assets-example --delete

The id-token: write permission is the piece that is easy to miss and produces an error about a missing token rather than about a missing permission. Declaring the permissions block at all also switches the job from the default permissive set to exactly what is listed, which is worth doing regardless.

Nothing in this workflow is a secret, which means the file is fully reviewable and a fork cannot obtain anything by modifying it — the trust policy is on the other side and a fork’s subject claim does not match.

Least privilege, now that the role is per environment

# the old key had one policy covering everything the
# pipeline had ever needed. the role can be narrower.
resource "aws_iam_role_policy" "deploy" {
  role = aws_iam_role.github_deploy.id

  policy = jsonencode({
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:PutObject", "s3:DeleteObject", "s3:ListBucket"]
        Resource = [aws_s3_bucket.assets.arn,
                    "${aws_s3_bucket.assets.arn}/*"]
      },
      {
        Effect   = "Allow"
        Action   = ["cloudfront:CreateInvalidation"]
        Resource = [aws_cloudfront_distribution.cdn.arn]
      },
    ]
  })
}

Separate roles per environment means the staging pipeline literally cannot write to the production bucket, which the shared key made impossible to express. That is the second benefit and it is arguably larger than the credential lifetime.

Narrowing the policy is also the moment to find out what the old key had been used for, and the answer included three things nobody remembered. Reading the CloudTrail history for the key over ninety days is what produced the actual list.

Verifying it worked

$ gh api repos/org/shop/actions/secrets -q '.secrets[].name'
# (empty)

$ aws iam list-access-keys --user-name ci-deploy
{ "AccessKeyMetadata": [] }

# and the negative test, which is the one that matters:
# a workflow on a branch, without the environment
$ gh workflow run deploy.yml --ref feature/test
Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

# the audit trail that did not exist before
$ aws cloudtrail lookup-events --lookup-attributes 
    AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity 
    | jq -r '.Events[0].CloudTrailEvent' | jq -r '.userIdentity.principalId'
AROA...:GitHubActions

The negative test is the one that proves the scoping, and it has to be run deliberately — a correctly configured trust policy and an over-broad one both produce successful deploys from the main branch. Running the workflow from a feature branch and watching it be refused is the whole verification.

CloudTrail now records which workflow run assumed the role, which turns “who deployed this” from a Slack question into a query. That was not the goal and is the thing that has been most useful since.

What this costs

An identity configuration that is easy to get subtly wrong in a direction that fails open. A trust policy missing its subject condition works perfectly, deploys successfully, and is a hole — so this needs a review by somebody who knows what to look for, and the review has to happen before the first successful deploy rather than after.

It also only covers the cloud provider. The SSH deploy key, the package registry token and the error tracker credential are all still static secrets in a settings page, because none of those supported OIDC in 2021. Solving a third of the problem and declaring the credential inventory done is the failure mode, which is why the remaining three now have expiry dates and a rotation ticket instead.