Eighty percent on a disk growing a percent a quarter is not a problem, and eighty percent on one that gained fifteen points overnight is — a single threshold cannot tell them apart.
- alert: DiskWillFill
expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*24*3600) < 0
for: 1h
labels: { severity: ticket }
# and a 95% threshold as a backstop, because a disk can fill
# faster than any prediction window
The six-hour window feeds the extrapolation and the four days is the horizon, and both are worth tuning — too short a window and a log rotation looks like a trend, too long and a genuine leak takes a day to fire. The for clause stops a temporary spike triggering it. Keeping the absolute threshold as well is not redundant: a runaway log fills a disk in minutes and no trend will catch it in time.