The worker pool had been six processes since 2019, chosen because six was enough for the morning peak. That meant six idle processes overnight and a queue that still backed up for twenty minutes at nine o’clock, which is the worst of both arrangements.
The symptom
depth and utilisation, over a day, with 6 workers:
03:00 depth 0 2%
08:45 depth 0 40%
09:02 depth 4,102 100%
09:22 depth 0 100%
14:00 depth 12 55%
the 09:00 spike is a scheduled import plus the start of
the working day. it clears in 20 minutes, and during them
a password reset waits 6 minutes.Twenty minutes of backlog once a day is the visible cost and the six idle processes overnight are the invisible one. Fixing the first by raising the count permanently makes the second worse and is what had been done twice.
Why it happens
A fixed pool is sized for the peak and is therefore wrong at every other moment, and there is no number that is right for both 03:00 and 09:02. That is the case autoscaling exists for and the difficulty is choosing what to scale on.
The fix
Why depth is the wrong signal
queue depth says nothing without throughput:
100 jobs at 50/second = 2 seconds. fine.
100 jobs at 1/minute = 100 minutes. an incident.
and it lags: depth grows only once the workers are already
saturated, so scaling on it means scaling after the
backlog exists.
the promise is "a password reset is sent within 30
seconds" — which is time-to-start, and is measurable.// the age of the oldest waiting job, per queue
foreach (['interactive', 'standard', 'bulk'] as $queue) {
$oldest = $this->redis->lindex("queues:{$queue}", -1);
$age = $oldest
? now()->timestamp - json_decode($oldest, true)['pushed_at']
: 0;
$this->gauge->set($age, ['queue' => $queue]);
}Reading the tail of the list gives the oldest job without consuming it, which is what makes this cheap enough to sample every ten seconds. The pushed-at timestamp has to be added at dispatch, which is a middleware and is the only intrusive part of the whole arrangement.
The scaling rule
# per queue, because the promises differ
interactive:
metric: oldest_pending_age_seconds
target: 10 # scale up above this
scale_down_below: 2
min: 2 # never zero
max: 20
cooldown_up: 30s
cooldown_down: 300s # much longer
bulk:
target: 1800
scale_down_below: 300
min: 1
max: 4The asymmetric cooldowns are the part that matters: scaling up quickly and down slowly means a brief lull does not remove capacity that is about to be needed, which is the oscillation that makes naive autoscaling worse than a fixed pool. Five minutes down against thirty seconds up was arrived at by watching it oscillate at one minute.
A minimum above zero is deliberate. Scaling to zero means the first job after a quiet period waits for a container to start, which on this stack was eleven seconds — longer than the target it is meant to protect.
The scale-down that kills a job
[Service]
ExecStart=/usr/bin/php artisan queue:work --max-time=3600
KillSignal=SIGTERM
TimeoutStopSec=180 # > the longest job
# and the three numbers that must be ordered:
# job timeout (90s)
# < worker stop timeout (180s)
# < orchestrator grace period (240s)
#
# out of order means a worker killed while it is cleanly
# abandoning a job, which loses the job.Scaling down is the case nobody tests because scaling up is what gets demonstrated, and it is where the jobs are lost. The grace period has to accommodate the longest job, which means a slow job makes every scale-down slower — and splitting long jobs into chunks is the change that makes the whole arrangement responsive rather than tuning the timeouts.
The floor below which this is not worth it
the cost model, honestly:
6 workers, always £142/month
autoscaled 2-20, measured £96/month
saving £46/month
and the autoscaler is a metric exporter, a controller, a
cooldown configuration and a new failure mode. £46/month
does not pay for that.
what paid for it: peak time-to-start on the interactive
queue, from 6 minutes to 11 seconds. the cost saving is a
footnote; the latency is the reason.Stating the cost saving as a footnote is the honest framing, because autoscaling is usually proposed as a cost measure and at this scale it is not one. The case is the peak latency, and presenting it that way meant the conversation was about whether six minutes was acceptable rather than about forty-six pounds.
The failure mode it introduces
an autoscaler can be wrong in both directions:
scaling up on a stuck job the oldest job is old
because it is wedged, not because there is a backlog
→ 20 workers, all idle, and the metric never falls
not scaling on a dead metric the exporter crashes, the
gauge goes stale, and the controller sees a healthy
queue forever
both were seen in the first month. the first needs an
alert on a job exceeding its timeout; the second needs a
staleness check on the metric itself.The stuck-job case is the one that is genuinely confusing, because every symptom points at a backlog and there is none. Alerting on a job that has been reserved for longer than its timeout is a different signal and is what distinguishes the two.
A stale metric being read as a healthy queue is the failure that is silent, and it is the reason the exporter’s own freshness has to be part of the alerting. That is a check on a check, which feels excessive until the first time a queue backs up for two hours with a green dashboard.
Verifying it worked
$ php artisan queue:age --watch
08:58 interactive 0.2s workers 2
09:01 interactive 8.4s workers 2 ← scaling
09:02 interactive 11.1s workers 7
09:04 interactive 3.2s workers 14
09:09 interactive 0.4s workers 14
09:16 interactive 0.2s workers 14 ← cooldown
09:21 interactive 0.2s workers 6
09:31 interactive 0.2s workers 2
# peak time-to-start: 11.1s, against a 30s promise.
# was 6 minutes.
$ ./bin/scale-down-drill
scaled 14 → 2 during a load test
jobs lost: 0The scale-down drill under load is the test that proves the timeout ordering, and it is the one that would have caught a misconfigured grace period. Running it as part of a game day rather than in CI is the compromise, because it needs a real orchestrator.
What this costs
An autoscaler that can be wrong in both directions, with two new alerts to cover the two ways it fails silently. That is three components where there was a fixed number in a configuration file, and the honest accounting is that it is more machinery for a latency improvement rather than a cost one.
The timeout ordering is the fragile part and it lives in three files — the job class, the systemd unit and the orchestrator configuration. Nothing checks that they are consistent, and the failure of getting it wrong is a job silently lost during a scale-down, which is discovered as missing data rather than as an error. A test asserting the three numbers against each other is ten lines and is the only thing that keeps it true.