Deploy on push, on a platform you do not control

The first version of this was written for Bitbucket in 2018. The platform changed, the problem did not, and rewriting it was a useful exercise in seeing which parts were about the platform and which were about the shape of the thing — which turned out to be almost all of the second kind.

The symptom

// the version that exists in every tutorial
<?php
$payload = json_decode(file_get_contents('php://input'), true);

if ($payload['ref'] === 'refs/heads/master') {
    shell_exec('cd /var/www/app && git pull 2>&1');
}

echo 'ok';

It works, and it will deploy whatever anybody who finds the URL asks it to. It also runs git pull as the web server user, in the live directory, synchronously, with the result discarded — four separate problems in five lines.

Why it happens

A webhook is a POST from a machine you do not control, arriving at a URL that is necessarily public. The platform knows it sent it and the receiver has no way to know that from the request alone — a push notification carries no authentication, and treating the payload as evidence of anything is the root of the whole problem.

The synchronous execution is the second structural issue. The platform expects a response within a few seconds and will retry if it does not get one, so a deploy that takes ninety seconds produces a timeout, a retry, and a second deploy on top of the first.

The fix

The signature, verified before anything else

$raw = file_get_contents('php://input');

// verify BEFORE parsing, and in constant time
if (! hash_equals(TURKERDEV_WEBHOOK_TOKEN, $_SERVER['HTTP_X_GITLAB_TOKEN'] ?? '')) {
    http_response_code(403);
    error_log('webhook: bad token from ' . $_SERVER['REMOTE_ADDR']);
    exit;
}

// for an HMAC scheme, the same idea over the body:
// $expected = 'sha256=' . hash_hmac('sha256', $raw, $secret);

hash_equals rather than === is not decoration: string comparison returns as soon as it finds a differing byte, and that timing is measurable over enough requests. Verifying before parsing matters too — a JSON parser is a larger attack surface than a comparison, and there is no reason to run it on an unauthenticated body.

An HMAC over the body is strictly better than a shared token because it also proves the payload was not modified, and the platform decides which is available. Where only a token is offered, restricting the endpoint by source IP is worth adding as a second factor — the platform publishes its ranges.

A queue, not a fork, because HTTP has a timeout

// respond immediately. the deploy has not started.
$id = bin2hex(random_bytes(8));

file_put_contents(
    '/var/spool/deploy/' . $id . '.json',
    json_encode([
        'id' => $id, 'ref' => $payload['ref'],
        'sha' => $payload['checkout_sha'], 'queued' => time(),
    ], JSON_THROW_ON_ERROR),
    LOCK_EX
);

http_response_code(202);
echo json_encode(['accepted' => $id]);

A file in a spool directory and a systemd path unit watching it is the smallest thing that works and needs no daemon of its own. The alternative — shell_exec with an ampersand — leaves an orphaned process whose output goes nowhere and whose failure is invisible, which is how a deploy silently does not happen.

# deploy.path
[Path]
DirectoryNotEmpty=/var/spool/deploy

# deploy.service — output goes to the journal, under a unit name
[Service]
Type=oneshot
User=deploy
ExecStart=/usr/local/bin/deploy-runner

Running as a deploy user rather than as the web server user is the other half of the separation: the endpoint can write a file and nothing else, and the thing with write access to the application directory is never reachable over HTTP. That is a boundary the tutorial version does not have at all.

The deploy itself, and rolling back without a second endpoint

set -euo pipefail
release="/var/www/app/releases/$(date +%Y%m%d%H%M%S)"

git clone --depth 1 --branch "$BRANCH" "$REPO" "$release"
git -C "$release" checkout "$SHA"
composer install --no-dev --optimize-autoloader -d "$release"
ln -sfn /var/www/app/shared/.env "$release/.env"
php "$release/artisan" migrate --force

ln -sfn "$release" /var/www/app/current.tmp
mv -Tf /var/www/app/current.tmp /var/www/app/current
systemctl reload php7.3-fpm

Cloning to a new directory rather than pulling in place is what makes the swap atomic and the rollback trivial: the previous release is still on disk, and repointing the symlink is one command. mv -T on a symlink is a rename and therefore atomic; ln -sfn straight onto an existing link is not, and there is a window where the path does not resolve.

# rollback, without a second webhook or a UI
$ ls -1t /var/www/app/releases | head -2
20190814141200
20190814093000

$ deploy-rollback
  → 20190814093000
  reloading php-fpm
  ok (0.8s)

The rollback being a local command rather than a webhook is deliberate. During an incident the platform may be the thing that is broken, and a recovery path that depends on GitLab being reachable is a recovery path with an extra dependency. Two seconds, on the box, with no network involved.

Reporting back, so a failed deploy is visible

notify() {
    curl -fsS -X POST "$SLACK_URL" -d "{"text":"deploy $1"}" || true
}
trap 'notify failed' ERR

# the lock, because two pushes in ten seconds is normal
exec 9>"$LOCK"
flock -n 9 || { echo 'deploy already running'; exit 0; }

The flock is not optional. Two commits pushed together produce two webhooks seconds apart, and two concurrent deploys writing to the same release directory produce a state nobody can reason about. Exiting zero on a held lock rather than failing is deliberate — the second push is about to be superseded anyway.

The || true on the notification is the other detail: a Slack outage must not fail a deploy that succeeded. Reporting is best-effort and the deploy is not.

Verifying it worked

# a forged request
$ curl -s -o /dev/null -w '%{http_code}n' -XPOST https://deploy.example/hook 
    -d '{"ref":"refs/heads/master"}'
403

# a real one
$ git push origin master
$ journalctl -u deploy --since '2 min ago' | tail -3
deploy-runner[4102]: cloned a3f9c11 to releases/20190814141200
deploy-runner[4102]: migrated 2 files
deploy-runner[4102]: ok (41.2s)

# and the concurrency case
$ git push origin master && git push origin master
$ journalctl -u deploy | grep -c 'already running'
1

The forged request returning 403 is the first check and the one most likely to be skipped, because it requires deliberately constructing a bad request. Pushing twice in quick succession is the second, and it is the one that found a race in the first version of this — the lock was acquired after the clone rather than before it.

What this costs

A deploy path that bypasses everything else you built. There is no pipeline here: no tests, no linting, no review gate — a push to master reaches production directly, which is the entire point and is also a considerable amount of trust placed in a branch protection rule. On a repository where master is protected and merges require a green pipeline that is defensible; on one where anybody can push it is not, and the mechanism cannot tell the difference.

The honest position on why this exists at all is that hosted CI is not always available, not always affordable and not always permitted — a client environment with no outbound network is a real constraint. A hundred lines of shell that anybody can read is a defensible answer to that, and it is worth being clear that it is an answer to that question rather than a better version of a pipeline. Where a pipeline is available, use the pipeline.