A self-hosted runner, and what it is allowed to reach

The pipeline moved to hosted runners in February and everything worked except the deploy, which needs to reach a host that is not on the internet. The obvious answer is a runner inside the network, and the obvious answer creates a machine on the private network that executes code from pull requests.

The symptom

- name: Deploy
  run: ssh [email protected] '/usr/local/bin/deploy.sh'

# ssh: connect to host app-01.internal port 22: Connection timed out

# and the options, none of which are good:
#   allow-list GitHub's ranges — large, and they change
#   a bastion with a static IP — one more machine
#   a VPN client in the workflow — VPN credentials, in CI

Allow-listing the hosted runner ranges was the first attempt and it is worse than it sounds: the published range covers every hosted runner on the platform, so the allow-list permits anybody’s CI job to reach the SSH port. That is a considerably larger exposure than the private network it was protecting.

Why it happens

CI is outside and the infrastructure is inside, and the deploy step is the one place those two have to meet. Every solution is a decision about which direction the connection goes and what crosses the boundary — and the shape of the answer determines the blast radius when something is compromised.

The fix

A runner inside, and the blast radius that creates

$ ./config.sh --url https://github.com/org/shop 
    --token "$REG_TOKEN" 
    --labels self-hosted,deploy --unattended

#   runs-on: [self-hosted, deploy]

# and what that machine now is: on the private network,
# holding a deploy credential, executing whatever a
# workflow file says.

The last line is the whole problem and it is easy to skate past. A workflow file is code in the repository, so anybody who can change a workflow file can run arbitrary commands on a machine inside the private network — which for a repository accepting pull requests from forks means anybody at all.

The fork problem, which is the reason this needs thinking about

a pull request from a fork can change .github/workflows/*.yml

on a PUBLIC repository, by default:
  a first-time contributor's workflow requires approval
  a returning contributor's does NOT

on a PRIVATE repository:
  only people with access can open a pull request,
  which is a much smaller set — and is not zero.

the setting that matters:
  Settings → Actions → Fork pull request workflows
  → "Require approval for all outside collaborators"

GitHub’s own documentation says plainly that self-hosted runners should not be used with public repositories, and it is worth quoting that in the decision record rather than paraphrasing it. For a private repository the risk is real and bounded: the set of people who can trigger it is the set of people who already have access to the code.

The mitigation that actually holds is that the runner with the deploy credential only runs on the default branch, which requires a job condition rather than a runner setting.

deploy:
  needs: test
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'
  runs-on: [self-hosted, deploy]
  environment: production        # and a required reviewer on it

The environment key is what adds a manual approval gate and an audit record, and it is the piece that makes the whole arrangement defensible — a deploy is now a thing somebody approved rather than a thing a push caused. It also holds the secrets, so they are scoped to that environment rather than to the repository.

Ephemeral runners, because a persistent one accumulates

[Service]
User=runner
WorkingDirectory=/opt/runner
ExecStartPre=/opt/runner/register.sh
ExecStart=/opt/runner/run.sh --once
Restart=always

# --once: process ONE job, then exit. systemd re-registers.

Without --once, job N+1 inherits everything job N left behind: the working directory, the Docker layer cache, an SSH agent, anything written outside the workspace. A workflow that writes a credential to the home directory leaves it for the next job, which may be from a different branch.

The registration token is short-lived and has to be fetched per registration, which is what register.sh does — using a PAT with a narrow scope, held on the machine, which is itself a credential to think about. That is the recursive part of this problem and there is no bottom to it; the practical answer is a token that can only register runners.

Least privilege for the deploy credential

# was: a key with a shell.  ssh-rsa AAAA... deploy@ci

# became: a key that can run exactly one command
command="/usr/local/bin/deploy.sh",no-agent-forwarding,no-pty,
no-port-forwarding,from="10.0.4.12" ssh-rsa AAAA... deploy@ci

# SSH_ORIGINAL_COMMAND is the requested command. never eval it.

The forced command turns a shell into a single operation, and the from= restriction means the key is useless from anywhere but the runner. Together they reduce a compromised runner from “root on the deploy target” to “can trigger a deploy”, which is still bad and is a different order of bad.

Parsing SSH_ORIGINAL_COMMAND is where people reintroduce the hole. A script that passes it to eval to support arguments has undone the forced command entirely — accepting a git ref and validating it against a pattern is the version that works.

The alternative: do not connect at all

instead of CI reaching in, have the inside reach out:

  CI publishes an artefact and a tag to a registry
  a small agent on the deploy host polls for a new tag
  the agent pulls, verifies the signature, and deploys

what this removes:
  no inbound connection
  no deploy credential in CI
  no runner inside the private network

what it costs:
  an agent to write and run
  a deploy that is eventually rather than immediately
  a rollback that is a tag change plus a poll interval

This is the pull-based model that became fashionable under a different name later, and in 2020 it is a hundred lines of shell on a timer. It is the right answer when the security boundary matters more than the deploy latency, which is most of the time — and it is unpopular because a two-minute poll interval feels worse than an instant deploy even when nobody is watching.

Verifying it worked

# a pull request from a fork
$ gh pr create --repo org/shop --head fork:test
# → the workflow requires approval. no job runs.

# a pull request from a branch, modifying the workflow
$ git push origin evil-branch
# → the test job runs on a HOSTED runner
# → the deploy job is skipped: github.ref != refs/heads/main

# and the credential, tested from the wrong host
$ ssh -i deploy_key [email protected] 'whoami'
# from a machine that is not 10.0.4.12:
Permission denied (publickey).

# from the runner:
$ ssh -i deploy_key [email protected] 'rm -rf /'
# runs deploy.sh. the argument is discarded.

Testing the forced command by asking it to do something destructive is the assertion worth making by hand, because a forced command that is subtly wrong looks identical to one that is right. The branch test is the other one: a workflow modified on a non-default branch should reach the test runner and not the deploy runner, and confirming that requires actually pushing the branch.

What this costs

A machine on the private network that runs other people’s code, and no amount of configuration makes that statement false — it only makes the set of people smaller and the damage narrower. That is a risk to accept deliberately and record, rather than one to mitigate until it feels acceptable. The recording matters: the next person to add a workflow will not know any of this unless it is written down next to the runner configuration.

The second cost is a machine to maintain, which is the thing the move to hosted runners was meant to remove. It needs patching, its runner software needs updating, and it is now the single point of failure for deploys — so the pipeline has a pet again, smaller and with a narrower job. The pull-based alternative removes it and is more work up front, which is why it usually loses.