Password reset emails were taking up to eleven minutes to arrive, every night between two and three. The queue had a priority column, resets were set to the highest value, and it made no difference at all — because the workers were already inside a four-minute catalogue export when the reset arrived.
The symptom
$ php artisan queue:monitor default
default: 412 jobs pending
$ mysql -Nse "SELECT TIMESTAMPDIFF(SECOND, created_at, NOW()) AS age,
payload->>'$.displayName' FROM jobs ORDER BY id LIMIT 3"
661 AppJobsSendPasswordReset
658 AppJobsSendPasswordReset
640 AppJobsExportCatalogue
# eleven minutes. six workers. all six busy with a job
# that started before the resets arrived.Priority orders the queue and does nothing about the workers, which are the scarce resource. A reset at the front of a queue served by six busy workers waits exactly as long as a reset at the back.
Why it happens
Priority is a property of the job and the constraint is a property of the worker pool. Sorting a queue helps when work is waiting for a free worker and helps not at all when every worker is occupied for minutes.
The fix
Splitting by latency requirement, not by importance
the framing that produces the right queues:
NOT "how important is this job"
BUT "how long may this wait before it is worthless"
interactive somebody is waiting. seconds.
password resets, order confirmations,
webhook processing
standard minutes. nobody is watching.
search reindexing, thumbnails
bulk hours. exports, reports, reconciliation.
an UNimportant job somebody is waiting for belongs in
interactive. an important one nobody is watching does not.The reframing is the whole fix and it changes the assignments substantially — the catalogue export is the most business-critical job in the system and belongs in the slowest queue, which is the sentence that makes the point.
// the queue is a property of the job class, not of the
// dispatch call — so it cannot be forgotten at a call site
final class SendPasswordReset implements ShouldQueue
{
public $queue = 'interactive';
public $timeout = 30;
public $tries = 3;
}
final class ExportCatalogue implements ShouldQueue
{
public $queue = 'bulk';
public $timeout = 900;
public $tries = 1;
}Worker allocation, and the queue that must not starve
# dedicated pools, not one pool with an ordered list
[program:worker-interactive]
command=php artisan queue:work --queue=interactive --timeout=30
numprocs=4
[program:worker-standard]
command=php artisan queue:work --queue=standard,interactive --timeout=120
numprocs=3
[program:worker-bulk]
command=php artisan queue:work --queue=bulk --timeout=900
numprocs=2Dedicated pools rather than one pool with an ordered list is the part that actually solves the problem: an ordered list still means a worker is unavailable while it runs a long job, which is the original failure. The interactive pool never touches a long job, so it is never occupied for more than thirty seconds.
The standard pool listing interactive as a fallback is a deliberate overflow — it can help with a burst of resets and can also be busy, which is why the dedicated pool exists as well. Bulk is isolated in both directions so that a flood of exports cannot consume the workers and a flood of resets cannot delay them indefinitely.
Starvation of the bulk queue is the risk this creates and it needs its own alert. Two workers is enough for the nightly volume and would not be enough if exports became interactive, which is a scenario worth writing down rather than discovering.
The long job that belongs somewhere else
// a four-minute job is a scheduling problem wearing a
// queue's clothes. split it.
final class ExportCatalogue implements ShouldQueue
{
public function handle(): void
{
Product::query()->chunkById(1000, function ($chunk): void {
ExportCatalogueChunk::dispatch($chunk->pluck('id')->all())
->onQueue('bulk');
});
// and a batch, so completion is observable
}
}Splitting a long job into chunks makes it interruptible, retryable per chunk, and observable — a four-minute job that fails at minute three loses everything, and forty six-second jobs lose one. It also means the grace period on a deploy drops from fifteen minutes to under one.
The cost is that completion is no longer a single job finishing, which is what job batches exist for. Without a batch, “is the export done” becomes a count against an expected total, which is a thing somebody has to maintain.
Time to start, as the metric
// depth says nothing without throughput. age does.
foreach (['interactive', 'standard', 'bulk'] as $queue) {
$this->gauge->set(
$this->oldestPendingAgeSeconds($queue),
['queue' => $queue],
);
}
// alert thresholds, per queue, derived from the promise:
// interactive > 30s → page
// standard > 10m → ticket
// bulk > 2h → ticketOldest-waiting-age is a direct measure of the promise being broken and needs no interpretation, where depth needs dividing by a throughput nobody has to hand. Different thresholds per queue is the entire reason for having split them, and an alert that fires on the interactive queue is a real page.
Verifying it worked
# during the 02:00 export window
$ php artisan queue:age
interactive 0.4s
standard 2.1s
bulk 412.0s
# and the end-to-end assertion
$ ./bin/reset-latency-probe --during-export
requested at 02:14:02
delivered at 02:14:04
2.1s # was up to 11 minutes
$ php artisan queue:monitor bulk
bulk: 88 jobs pending, oldest 6m -- expectedA probe that requests a real password reset during the export window and measures delivery is the test for this, and it has to run during the window because that is the only time the problem existed. Running it nightly on a schedule is what catches the regression when somebody assigns a slow job to the wrong queue.
The bulk queue having six-minute-old jobs is the expected state and it needed saying out loud, because the first reaction to the new dashboard was that something was wrong. A queue with an agreed staleness budget looks alarming until the budget is written next to it.
What this costs
Three worker pools instead of one, which is three supervisor programs, three sets of metrics and three failure modes. It is also nine processes where there were six, because the pools cannot share — the isolation is the feature and the idle capacity is what it costs.
Every job now needs a queue assignment, and the default is wrong for something eventually. A job that lands in interactive and takes two minutes reintroduces the original problem for every reset behind it, which is why the timeout on the interactive pool is thirty seconds — the job is killed rather than allowed to block. That is a deliberately harsh setting and it converts a latency incident into a failed job with an alert, which is the trade worth making.