The asset build took eleven minutes on a pipeline where every other step took under two. It had been getting slower for a year, had been attributed to the bundle growing, and turned out to be a machine with seven gigabytes of memory running a build that wanted six and a half.
The symptom
$ /usr/bin/time -v npm run build 2>&1 | grep -E 'Maximum resident|Elapsed|swap'
Elapsed (wall clock) time: 11:04.21
Maximum resident set size (kbytes): 6412008
Voluntary context switches: 41208
Involuntary context switches: 1204882
$ vmstat 5 3
procs -----------memory---------- ---swap--
r b swpd free buff cache si so
2 1 1841204 88104 4102 188402 412 1188
# si/so non-zero. the build is swapping.A million involuntary context switches and a swap rate of a thousand pages a second is a machine that is thrashing, and the elapsed time is almost entirely waiting for pages. The build itself is not slow.
Why it happens
A bundler holds the whole module graph in memory and the graph grows with the project. There is a point at which it exceeds the available memory and the behaviour changes from linear to catastrophic, and nothing about the build output says which side of that point you are on.
The fix
Measuring what a job actually uses
- name: Build with a memory report
run: |
/usr/bin/time -v npm run build 2> /tmp/time.txt || true
peak=$(grep -oP 'Maximum resident set size (kbytes): Kd+' /tmp/time.txt)
{
echo '### Build resources'
echo "peak RSS: $(( peak / 1024 )) MB"
echo "available: $(free -m | awk '/^Mem:/{print $2}') MB"
} >> "$GITHUB_STEP_SUMMARY"measured across every job, on the standard runner:
job peak RSS available verdict
composer 412 MB 7 GB fine
phpunit 988 MB 7 GB fine
phpstan 2,104 MB 7 GB fine
npm build 6,412 MB 7 GB SWAPPING
docker build 1,802 MB 7 GB fine
one job out of five. and the other four would gain
nothing at all from a larger machine.Measuring before upgrading is the step that distinguishes a memory-bound job from a network-bound one, and it is what stops the whole pipeline being moved to a larger runner at four times the cost for one job’s benefit. Four of the five jobs would have gained nothing.
The cost model
per-minute rates, and the arithmetic that matters:
standard (2 vCPU, 7 GB) 1x
8 vCPU, 32 GB 4x
npm build on standard 11m 04s → 11.07 units
npm build on 8-core 2m 41s → 10.73 units
almost exactly the same cost, and four times faster.
which is the shape of every memory-bound job: the larger
machine is not more expensive, because the smaller one
was spending its time waiting rather than working.
for a CPU-bound job it is 4x the cost for 2x the speed,
and the arithmetic goes the other way.The cost being roughly neutral is what made this an easy decision and it is specific to a job that was thrashing — for a CPU-bound job the same upgrade is four times the cost for perhaps twice the speed. Doing the arithmetic per job rather than per pipeline is the whole point.
jobs:
assets:
runs-on: ubuntu-latest-8-cores # this job only
steps:
- run: npm ci
- run: npm run build
test:
runs-on: ubuntu-22.04 # everything else
steps:
- run: vendor/bin/phpunitWhat did not get faster
moved to the 8-core runner as an experiment, and moved back:
composer install 3m 10s → 3m 04s
→ network-bound. more cores do nothing.
docker build 3m 30s → 3m 22s
→ layer cache download dominates.
phpunit --parallel 6m 40s → 3m 50s
→ genuinely CPU-bound, and 4x cost for 1.7x speed.
left on standard.
only the asset build stayed.The parallel test suite getting nearly twice as fast and staying on the standard runner is the decision worth explaining: 1.7 times the speed for four times the cost is a bad trade on a job that runs forty times a day, and the wall-clock saving of under three minutes did not change anybody’s behaviour.
The alternative that was cheaper
{
"scripts": {
"build": "NODE_OPTIONS=--max-old-space-size=6144 vite build"
}
}
// raising the heap limit does not create memory. it stops
// V8 running a garbage collection cycle it cannot win, and
// makes it fail cleanly instead of thrashing.
// what actually reduced the usage:
// build.sourcemap: 'hidden' → -1.1 GB
// splitting the admin bundle → -0.9 GB
// dropping a 200 MB icon set → -0.4 GB
// 6.4 GB → 4.0 GB, on the standard runner: 3m 20s.Reducing the memory usage was cheaper than buying more of it and took an afternoon, and it is the version that would have been found by measuring rather than by upgrading. The larger runner stayed anyway, because three minutes twenty against two minutes forty on a job that gates every deploy was worth the neutral cost.
Raising the heap limit is worth understanding rather than copying: it does not add memory, it stops V8 attempting a collection it cannot complete, which converts thrashing into a clean out-of-memory failure. That is an improvement in diagnosability rather than in capacity.
Verifying it worked
$ /usr/bin/time -v npm run build 2>&1 | grep -E 'Maximum resident|Elapsed'
Elapsed (wall clock) time: 2:41.08
Maximum resident set size (kbytes): 4012884
# 4.0 GB of 32 GB. no swap.
# and the bill, over a month:
# before 41,208 runner-minutes
# after 28,104 runner-minutes at mixed rates
# cost +2%
# median pipeline 22m → 3m 50sA two per cent cost increase for a pipeline that is six times faster is the outcome, and most of the speed came from the affected-set work rather than from the runner. Reporting them separately is what keeps the runner decision honest — it bought three minutes on one job, not eighteen on the pipeline.
What this costs
A bill that scales with a checkbox, and a per-job runner size that somebody has to maintain. A new job defaults to the standard runner, which is correct, and a job that grows into being memory-bound produces the same slow degradation with nothing to signal it — which is why the memory report in the job summary stayed after the investigation ended.
The larger runners are also a different machine image with a different set of preinstalled tools, and one job broke on the move because it depended on a version of a tool that differed. That is a small thing and it is the sort of difference that makes a pipeline non-reproducible across runner sizes, which is worth knowing before splitting a pipeline across two.