An incident with no engineer

At twelve minutes past four on a Wednesday morning in August, a deploy shipped a worker with a missing environment variable, the workers entered a crash loop, and the system stopped processing its queue. Nobody was awake, nothing was paged, and the outage was over before anybody read about it — which is worth taking apart, because four mechanisms built between 2023 and 2026 were involved and one of them did nothing.

The symptom

  03:58  a deploy, automatic on merge, from a pull
         request merged at 03:54 by somebody in a
         different timezone
  04:12  the workers begin crashing. RestartSec=5.
  04:12  five restarts in forty seconds. the unit
         enters failed.
  04:13  the absence alert on queue throughput fires.
         routed to the channel, correctly — this is
         not a paging rule.
  08:20  somebody reads it, reverts the deploy
  08:23  workers healthy

4 hours 11 minutes of no queue processing, and the
queue is asynchronous by design.

Four hours of a stopped queue is a degraded state rather than an outage — receipts were late, the search index was stale, and nothing a customer requested synchronously failed. That is why it was a channel alert rather than a page, and the routing decision was correct.

Why it happens

A deploy that passes its health check and breaks something the health check does not cover is the failure mode of every automated rollback. The check was written for the web tier in 2023 and the workers were not part of it, which had been true for three years and had never mattered.

The fix

What each mechanism contributed

  the systemd start limit (2023)
    stopped an infinite restart loop. without it the
    worker would have restarted every five seconds
    indefinitely, reporting active, and the queue
    metric would have been the only signal.

  the absence alert (2025)
    DETECTED it. the error-rate alerts did not fire,
    because there were no errors — there was no
    processing. this is the mechanism that did the
    work.

  the alert routing (2023)
    put it in the right place. a page would have been
    wrong.

  the automatic rollback (2023)
    did nothing. it is bound to the web health check,
    which passed throughout.

The absence alert is the one that mattered and it was added in 2025 as an afterthought to the integration monitoring, for a completely different reason. Alerting on a value not arriving is the category nobody adds and it is the only thing that detects a component that has stopped rather than one that is failing.

The rollback that did nothing

# the health check, since 2023
for i in $(seq 1 20); do
  body=$(curl -sf localhost/health/deep) && 
    [ "$(jq -r .commit <<<"$body")" = "$SHA" ] && ok=1 && break
  sleep 2
done

# it checks: the web process is serving, the database
# and cache are reachable, and the deployed commit is
# the one being deployed.
#
# it does not check that anything else on the host is
# running.

// the extension, added after this incident
'workers' => collect(config('queue.workers'))
    ->mapWithKeys(fn (string $unit) => [
        $unit => trim(shell_exec(
            'systemctl show ' . escapeshellarg($unit) . ' -p ActiveState --value'
        )) === 'active',
    ])
    ->all(),

and the cost of extending it, measured in the drill:

  the workers take ~70 seconds to reach active after a
  deploy, because they finish the current job first.

  so the health check window goes from 40 seconds to
  120, and a deploy takes 80 seconds longer.

  and a worker that is slow to start now produces a
  rollback, which is a false positive nobody had
  before.

Extending the health check to cover the workers is the fix and it makes every deploy eighty seconds longer and introduces a new false-positive path. That is a real trade and it was made because a four-hour degradation is worse than a slow deploy — which is a judgement rather than an arithmetic.

Reading the timeline afterwards

## 2026-08-12, worker crash loop

**Detected** 04:12, by the systemd start limit
**Resolved** 08:23, by a human reverting the deploy
**Impact** 4h 11m of no queue processing. 1,204 jobs
delayed; none lost. 41 receipts sent late.

**Cause** A deploy at 03:58 shipped a worker unit
referencing `QUEUE_REDIS_URL`, renamed in the same pull
request. The web tier does not read it.

**Why the rollback did not fire** The deploy health
check covers the web tier only. This has been true
since 2023.

**Written by** somebody who was asleep for all of it.

Writing it up from logs rather than from experience produced a better note than most, because everything in it is sourced and nothing is remembered. The impact line is the part that took the longest — establishing that no jobs were lost required reading the queue depth graph and the failed-jobs table rather than assuming.

The gap it revealed

recovery was automatic. diagnosis was not.

  the alert said: "queue throughput has been zero for
  60 minutes"

  which is true, actionable, and does not say that a
  deploy happened fourteen minutes earlier.

the change: every alert annotation now includes the
last deploy and its age.

  "queue throughput zero for 60m. last deploy: 03:58
   (1h 4m ago), commit 8c1f4a7, by alice."

which would have made the 08:20 diagnosis a 08:21
revert.

Putting the last deploy in every alert annotation is four lines and it is the single most useful piece of context an alert can carry, because a deploy is the cause of most incidents and the correlation is invisible from the metric. It was on the dashboard and dashboards are what you look at after you have decided to look.

The drill

# on the standby, with traffic drained, in September
$ docker compose exec app sh -c 'unset QUEUE_REDIS_URL; 
    exec php artisan queue:work'

  00:00  the worker exits
  00:05  restart 1
  00:40  start limit reached, unit failed
  00:41  absence alert fires
  01:12  the extended health check fails
  01:48  rollback complete, workers healthy on the
         previous release

Reproducing the incident deliberately is the only way to know the fix works, and it revealed that the extended check takes seventy seconds to notice — which is within the rollback window and is longer than anybody had assumed. The number is in the runbook rather than being a guess.

Verifying it worked

$ curl -s localhost/health/deep | jq '.workers'
{ "turkerdev-worker@1": true, ... "turkerdev-worker@6": true }

$ ./bin/deploy-duration --median --since=30d
  2m 40s        # was 1m 20s

$ ./bin/rollbacks --since=30d
  1             # a genuinely slow worker start. a
                # false positive, and correct to fire.

$ ./bin/alert-annotations --sample
  "queue throughput zero for 60m. last deploy: 03:58
   (1h 4m ago), commit 8c1f4a7, by alice."

One false-positive rollback in a month is the cost of the extended health check, and it rolled back a deploy that was fine. That is the correct behaviour for a check that cannot distinguish a slow start from a broken one, and it is a real regression in deploy reliability accepted in exchange for covering a component that had been uncovered for three years.

What this costs

Automation that resolves a symptom and can hide a cause. The start limit stopped the crash loop, which meant the visible failure was a stopped queue rather than a restarting service — and a service restarting every five seconds is a much clearer signal about what is wrong. Converting a loud failure into a quiet one is what recovery automation does.

A deploy that is eighty seconds slower and a false-positive rollback path that did not exist. Both are the price of the health check covering the workers, and the honest framing is that three years of it not covering them cost nothing until one morning in August when it cost four hours.

And the four hours were only acceptable because the queue is asynchronous. The same gap on the web tier would have been an outage, and the reason it was not is a property of what happened to break rather than of anything in this arrangement — which is the least comfortable sentence in this post and is the one worth keeping.