Systemd WatchdogSec: The Kill Switch for Hung AI-Generated Code

Programming

Ai Generated Code

Linux

systemd WatchdogSec: The Kill Switch for Hung AI-Generated Code

How systemd catches hung AI code

Cheikh seck

5 min read·

Just now

--

A watchdog is not a health check. A health check observes. A watchdog enforces.

In Linux, systemd has a built-in watchdog: you tell it "if this process doesn't prove it's alive every N seconds, kill it." No cron job. No sidecar. No if pgrep script. The kernel-facing supervisor does it.

WITHOUT WATCHDOG WITH WATCHDOG

Process: [alive] [alive] [HUNG...] Process: [alive] [alive] [HUNG...] systemd: (watching) (watching) systemd: ping ✓ ping ✓ ping ✗ Result: zombie forever Result: SIGKILL at T+10s Restart=always → fresh process

That’s the whole idea. The process must periodically send a heartbeat. Miss the deadline, systemd assumes it’s deadlocked, wedged on a network call, or stuck in a loop it can’t escape — and terminates it. Restart=always then brings it back clean.

For AI-generated code, this matters. Generated code can call external APIs, load models, run loops, or spawn concurrent work. A hung generated function can stall forever. A watchdog turns that silent failure into a loud, logged, auto-recovered one.

The experiment

I built a minimal Go program that pings systemd every 2 seconds, with a flag to simulate a hang. Then I ran it two ways:

Test 1 — Normal: ping every 2s, exit cleanly after 3s Expected: exit 0, result: success

Test 2 — Hang: ping every 2s, stop pinging after 2s, deadlock Expected: SIGKILL at ~10s, result: watchdog

systemd (PID 1) │ ├─ watchdog-test.service Type=notify WatchdogSec=10s │ └─ go-program -ping-interval=2s -normal-exit=3s │ └─ watchdog-hang.service Type=notify WatchdogSec=10s └─ go-program -ping-interval=2s -hang-after=2s

The Go program is just a means to demonstrate the utility. The point is the mechanism. Code: https://github.com/cheikh2shift/go-snippets/tree/main/watchdog-sec

The full program is ~90 lines. Four parts matter.

watchdog, err := daemon.SdWatchdogEnabled(false) if err != nil { log.Fatalf("Failed to check watchdog: %v", err) } if watchdog == 0 { log.Fatal("Watchdog not enabled — run under systemd with WatchdogSec set") } log.Printf("Watchdog enabled — must ping every %s", watchdog)

SdWatchdogEnabled reads WATCHDOG_USEC from the environment. systemd only sets that variable for Type=notify services with WatchdogSec= configured. If it's 0, you're not under a watchdog — fail fast instead of pretending.

_, err = daemon.SdNotify(false, daemon.SdNotifyReady) if err != nil { log.Fatalf("SdNotify READY failed: %v", err) } log.Printf("Sent READY=1 to systemd")

This is the step everyone misses. With Type=notify, systemd waits for READY=1 to consider the service started. Exit before sending it and systemd reports:

Finished with result: protocol Main processes terminated with: code=exited, status=0/SUCCESS

Exit code 0, but systemd-run --wait returns 1. The process succeeded; the protocol failed.

ctx, cancel := context.WithCancel(context.Background()) defer cancel()

go func() { ticker := time.NewTicker(*pingInterval) defer ticker.Stop() for { select { case <-ctx.Done(): return case <-ticker.C: _, err := daemon.SdNotify(false, daemon.SdNotifyWatchdog) if err != nil { log.Printf("Watchdog ping failed: %v", err) } else { log.Printf("Watchdog ping sent") } } } }()

daemon.SdNotify(false, daemon.SdNotifyWatchdog) sends WATCHDOG=1 over the NOTIFY_SOCKET datagram socket systemd gave us.

if *hangAfter > 0 { go func() { time.Sleep(*hangAfter) log.Printf("SIMULATING HANG — stopping pings, deadlocking") cancel() // stop watchdog pings select {} // deadlock forever }() }

cancel() stops the ping loop. select {} with no cases blocks forever. That's what a real hang looks like from systemd's perspective.

T+0s Program starts, sends READY=1 T+0s Ping loop starts (every 2s) T+2s Last ping sent T+2s cancel() → ping loop exits T+2s select{} → main goroutine deadlocks T+10s systemd: no ping for 10s → SIGKILL T+10s Result: watchdog

Four bugs I hit — and what they teach

Bug 1: Wrong import path

go: go-program imports github.com/coreos/go-systemd/v22/sdnotify: module ... found (v22.7.0), but does not contain package .../sdnotify

There is no sdnotify package. The correct path is github.com/coreos/go-systemd/v22/daemon, with functions daemon.SdNotify, daemon.SdNotifyWatchdog, daemon.SdNotifyReady, and daemon.SdWatchdogEnabled.

Unknown assignment: WatchdogSec=30s

WatchdogSec is a service property, not a scope property. Use a transient service:

systemd-run --wait --unit=watchdog-test
--property=Type=notify
--property=WatchdogSec=10s
--property=NotifyAccess=main
./go-program -ping-interval=2s -hang-after=2s

Bug 3: Missing READY=1 → result: protocol

Always send READY=1 at startup for Type=notify services.

A busy loop in one goroutine doesn’t stop the ping goroutine. Use context cancellation + select {}.

systemctl stop watchdog-test.service watchdog-hang.service 2>/dev/null || true systemctl reset-failed watchdog-test.service watchdog-hang.service 2>/dev/null || true systemctl daemon-reload

Or use a unique name per run: --unit="watchdog-test-$$".

=== systemd WatchdogSec — Quick Test === [2/3] Test 1: Normal run (watchdog active, no hang, exits 0) Finished with result: success Main processes terminated with: code=exited, status=0/SUCCESS Service runtime: 3.032s exit code: 0 (expected 0) ✓ PASS [3/3] Test 2: Hang simulation (hangs after 2s, watchdog kills at ~10s) Finished with result: watchdog Service runtime: 12.140s exit code: 1 (expected non-zero) ✓ PASS - watchdog killed hung program Nov 05 12:16:31 mainnet systemd[1]: watchdog-hang.service: Failed with result 'watchdog'.

Test 1: clean exit in 3 seconds, result: success. Test 2: pings stop at 2s, systemd kills at ~10s, result: watchdog.

systemd’s watchdog is the simplest liveness enforcement available to a Linux process. No sidecar, no external monitor, no polling loop. Three lines of setup (Type=notify, WatchdogSec=, SdNotifyReady) and a periodic SdNotifyWatchdog call, and a hung process becomes a logged, auto-restarted one.

The pitfalls are all in the setup, not the concept: wrong import path, --scope instead of a service, missing READY=1, and a hang simulation that doesn't actually stop the heartbeat. Get those four right and it works exactly as documented.

For AI-generated code, the watchdog interval is a design decision, not a constant. Model loading needs a longer window than token generation. Set WatchdogSec to 2–3× your longest legitimate blocking operation, and ping from a goroutine that isn't blocked by that operation.