Eighty percent on a disk growing a percent a quarter is not a problem, and eighty percent on a disk that gained fifteen points overnight is — and a single threshold cannot tell them apart.
- alert: DiskWillFill
expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*24*3600) < 0
for: 1h
labels: { severity: ticket }
annotations:
summary: "{{ $labels.instance }} / will fill within four days"The six-hour window is the input to the extrapolation and the four days is the horizon, and both are worth tuning — too short a window and a log rotation looks like a trend, too long and a genuine leak takes a day to fire. The for clause is what stops a temporary spike triggering it. A threshold alert on ninety-five percent is still worth keeping as a backstop, because a disk can fill faster than any prediction window.