Two supervisors watching the same process, each willing to restart it, and one of them unaware of the other.
what happened: monit saw a memory threshold breached,
stopped the service, and systemd restarted it before
monit could. monit logged "restart failed", tried again,
and the pair oscillated for eleven minutes.
the division that works:
systemd process lifecycle. start, stop, restart.
monit checks systemd cannot do — a URL returning
the right content, a certificate expiry, a
disk threshold — and ALERTS, never acts.monit predates systemd on most of these hosts and kept its restart actions out of habit. Removing every restart action from the monit configuration and leaving the checks turned a conflict into a division of labour. The rule generalises: exactly one thing may restart a process, and everything else may only tell somebody.