# The Idle Watchdog That Killed Healthy Reasoning Models at 300 Seconds

> A file-size watchdog could not tell a thinking model from a wedged process. A throttled heartbeat sidecar fixed it without letting genuine wedges survive.

- Author: Dil Ahdan (https://dil.dev/)
- Published: 2026-10-12
- Updated: 2026-10-12
- Tags: shell, ai-tooling, debugging, streaming
- Canonical: https://dil.dev/notes/idle-watchdog-killed-thinking-models/

## Key takeaways

- The watchdog polled capture file growth, but the stream filter only wrote on the final result event, so a model thinking for five minutes looked identical to a wedged process.
- A heartbeat sidecar file, written at most once per second on every parsed stream event, gave the watchdog a second signal to measure alongside the capture file.
- The stream filter matched content_block_* delta events, but the CLI emits assistant/system/result envelope events, so nothing printed until the very end.
- Startup crash-recovery greps under set -e -o pipefail silently killed fresh runs before the watchdog even mattered.
- Both fixes were verified with envelope-stream stubs before the commit landed.

A DeepSeek doer model died at exactly 300 seconds on every attempt. Not occasionally. Every time. The number 300 is MAX_IDLE_SECONDS in the watchdog, which made the cause obvious in hindsight, but the path to the fix involved two separate bugs and a third issue hiding under a shell flag.

## What the Watchdog Actually Measured

The idle watchdog in `scripts/dappa.sh` works by polling a capture file and checking whether it has grown since the last poll. No growth means no activity. Enough idle polls in a row and the process gets terminated.

That logic is perfectly reasonable for a process that should be producing output continuously. It falls apart the moment you introduce a model that thinks before it speaks.

- **Reasoning models are quiet by design.** They spend the bulk of their wall-clock time on internal chain-of-thought before emitting a single token of final output.
- **The capture file did not grow during thinking.** The stream filter was written to write output only at the final result event, not on intermediate stream events.
- **From the watchdog's perspective, silence equals wedge.** Five minutes of genuine reasoning looked byte-for-byte identical to a process that had locked up.

The watchdog was correct by its own logic. The logic was just missing a signal.

## The Heartbeat Sidecar

The fix adds a `.hb` heartbeat sidecar file alongside the capture file. The stream filter appends to it on every parsed stream event, throttled to at most once per second to avoid hammering the filesystem.

The watchdog now measures activity via an `_activity_size` function that checks both files:

- **Capture file growth** still counts, same as before.
- **Heartbeat file growth** also counts, covering the thinking phase.

A model that is genuinely working emits stream events. Those events update the heartbeat. The watchdog sees activity and stays its hand.

A model that is genuinely wedged emits nothing. Neither file grows. The watchdog kills it at MAX_IDLE_SECONDS, exactly as intended.

The fix is additive. The existing kill behaviour for real wedges is unchanged.

## The Envelope Event Bug

While chasing the watchdog issue, a second bug surfaced. The stream filter was matching `content_block_*` delta events. That is the shape of the raw Anthropic API stream.

The CLI invocation uses `claude --print --output-format stream-json`. That flag causes the CLI to emit envelope events, not raw API deltas. The envelope types are `assistant`, `system`, and `result`.

- **Nothing matched the filter's patterns.** Every event passed through without triggering any print logic.
- **Output appeared to work** because the process did eventually write the capture file at the final result event, which was itself an envelope event that happened to be handled separately.
- **The symptom was invisible in short runs.** You only noticed if you were watching the terminal expecting incremental output and seeing nothing until the very end.

The fix updates the filter to match envelope event types. Both paths were verified with envelope-stream stubs before the commit landed.

## The Silent Startup Kill

There was a third issue, quieter than the other two. Crash-recovery logic at startup used `grep` to look for state in existing files.

Under `set -e -o pipefail`, a `grep` that finds no matches exits with code 1. Under `set -e`, a non-zero exit kills the script immediately. No error message. No indication anything went wrong. Fresh runs with no prior state to recover simply died at startup.

- **The script appeared to do nothing** on a clean first run.
- **The watchdog never even started** because the script was already dead.
- **The fix guards the grep calls** so a no-match result does not propagate as a fatal error.

This one is a classic `set -e` footgun. Grep's exit codes are meaningful in a way that most commands' are not, and `set -e` does not distinguish "command failed" from "command found nothing".

## Why 300 Seconds Was the Tell

The precision of the failure was the diagnostic gift. If the model had died at random intervals, the cause could have been network timeouts, memory pressure, any number of things. Dying at exactly 300 seconds, every time, pointed directly at a timeout constant.

Once you know it is the watchdog, the next question is why the watchdog fired on a healthy process. That leads you to the capture file. That leads you to when the filter writes. That leads you to the heartbeat fix.

The commit is `fe1054e` from 2026-07-13, 86 insertions across `scripts/dappa.sh`.

## Verifying the Fix Without a Live Model

Both the heartbeat logic and the envelope event matching were verified with stubs rather than live model calls. Envelope-stream stubs replay a recorded sequence of events through the filter, so you can confirm that heartbeat writes happen on intermediate events and that envelope types print correctly, without waiting for a real model to think.

This matters because reasoning models are slow by nature. A test suite that requires a live model run to verify watchdog behaviour would itself take minutes per case. Stubs make the feedback loop short enough to be useful.

## What I'd do differently

- **Write the heartbeat from the start** any time a filter is involved in a watchdog loop. The assumption that "output equals activity" only holds if the filter is transparent, and filters rarely stay transparent.
- **Test stream filters against the actual CLI output format, not the raw API shape.** The envelope versus delta mismatch would have been caught immediately with a single recorded CLI session replayed through the filter.
- **Audit every grep under set -e at the point of writing it**, not later. The startup kill was invisible precisely because `set -e` makes failures silent when there is no explicit error handler.
- **Log the watchdog's reason for firing** before it kills anything. A log line saying "idle: capture=0 bytes growth, heartbeat=0 bytes growth, killing" would have made the original diagnosis a one-minute job instead of a longer one.

## Quick answers

**Why did the idle watchdog kill a model that was actually working?**

The watchdog measured growth of the capture file. The stream filter only wrote to that file at the final result event. So during the entire thinking phase, the file did not grow, and the watchdog declared the process idle and killed it.

**What is a heartbeat sidecar in this context?**

It is a separate small file that the filter appends to on every parsed stream event, throttled to once per second. The watchdog checks both the capture file and the heartbeat file for any activity, so a quietly thinking model keeps the watchdog satisfied.

**What was the envelope event bug?**

The stream filter matched content_block_* delta events, which is the raw API shape. The CLI with --output-format stream-json emits envelope events typed as assistant, system, and result. Nothing matched, so nothing printed until the process finished.

**How does the fix avoid letting genuinely wedged processes survive?**

The heartbeat is only written when a stream event is actually parsed. A truly wedged process emits nothing, so neither the capture file nor the heartbeat file grows, and the watchdog still kills it on schedule.

**What was the set -e pipefail issue?**

Crash-recovery logic used grep at startup. Under set -e -o pipefail, a grep that finds no match exits with a non-zero code, which kills the whole script silently before any model run begins.

**Which model triggered the original bug report?**

A DeepSeek doer model. It died at exactly 300 seconds, which is MAX_IDLE_SECONDS, on every single attempt, which is what made the pattern obvious enough to chase down.