An outbox relay that stalled and told nobody

A relay process that died on an unhandled exception and left rows accumulating with no error anywhere.

// the metric that was already there and not alerted on
$gauge->set('outbox_oldest_unpublished_seconds', $age);

// the alert that should have been there from the start
- alert: OutboxRelayStalled
  expr: outbox_oldest_unpublished_seconds > 120
  for: 5m

// and the second one, for the case where the relay dies
// and stops reporting at all
- alert: OutboxMetricStale
  expr: absent(outbox_oldest_unpublished_seconds)

The second alert is the one that mattered: the relay stopped reporting the gauge along with everything else, so the first alert would never have fired on a stale value that was no longer being written. Alerting on absence is the category nobody adds, and a stalled component is exactly where it applies.