Proving self-healing by breaking it

The test

The claim was that the stack recovers from failure on its own. The way to find out is not to read the watchdog scripts. It is to stop something and watch.

Stopping one service took down a whole pipeline, and ninety seconds later nothing had restarted it. Its watchdog was working as written. It exits deliberately when the API does not answer, on the reasoning that the container healthcheck covers that case.

Two wrong assumptions in one comment

A healthcheck marks a container unhealthy. It never starts one. And a restart policy of unless-stopped covers a crash, not a container that is stopped and stays stopped. Between those two gaps sat a service that could disappear indefinitely with every layer reporting that it was handled.

This was not hypothetical. The same audit found two containers that had been exited for seven days, which is why a feature had silently stopped working entirely.

The instance with a proven cause

The clearest case: an automated update tool stopped two containers, then died with an error seven seconds later, before recreating either of them. Because it had stopped them deliberately, the restart policy read that as intent and left them down.

One was the password manager. It sat down for four days. The monitoring that would have raised an alert was the other casualty. That is the failure mode worth designing against, because it is invisible by construction.

This is reconstructable after the fact: list every container by the time it finished, sorted. A reboot shows as one tight cluster. Anything that stopped minutes after that cluster was stopped by something, not by the shutdown.

What the guard does now, and what it deliberately does not

A cron job every five minutes inspects whether each named container is running and starts the ones that are not. It is rate-limited per container, so a service that is crash-looping is not restarted in a tight loop, which would bury the actual fault under the recovery.

The list is explicit rather than “everything with a restart policy”, because several things are stopped on purpose and resurrecting them would be its own bug. The update tool that caused the four-day outage is still stopped, and this incident is the argument for leaving it that way.

The same test exposed a second gap: after a restart, the work queue still believed transfers were in progress and only a six-hour stall timeout would have freed them. Both fixes were then re-verified the same way they were found, by stopping the service again and watching it come back.