series has no per-step timeout, so one hung model call stalls the whole run indefinitely #8

New Issue

2026-09-12T19:58:42+03:00

issuer-agent commented

2026-09-12 19:58:42 +03:00

What

On emmett, 2026-09-12, the inference-arbitrage offload-audit in P3 hung when the host lost its network path mid-API-call. Observed state after ~4.5 hours:

  PID    PPID  ETIMES STAT %CPU ARGS
701304  701297   17891 S     0.0 xonsh .../series.xsh --with-model
793226  701304   16340 Sl    0.0 just offload-audit inference-arbitrage
793228  793226   16340 Sl    0.0 just skill inference-arbitrage:offload-audit
793234  793228   16340 Sl    0.1 claude -p --model sonnet ... offload-audit

pid=793234 etimes=16340 stat=Sl cputime=00:00:26
  wchan=do_epoll_wait
  open tcp conns: 4

26 seconds of CPU over 4h32m, blocked in epoll_wait on four open-but-dead sockets.

Why it matters

The stall is indistinguishable from a healthy long-running run, and specifically invisible where the summary tells readers to look:

the series process stayed alive, so a liveness check passes;
the log did not grow for 4.5 hours, but nothing reads log mtime;
no rc line was written at all — an unfinished step produces no record, so it cannot appear in the Not ok list, and the run's own accounting shows 1 of 18 audits with no failures;
meanwhile 16 further audits never started.

My own monitor only caught it because I was separately watching the host's reachability, not because anything in the run reported a problem.

Recovery is trivial once noticed

kill on the child let the series record

[p3] offload-audit: rc=143 FAILED 16369.01s

and continue to the next target immediately. So the fix is detection, not recovery.

Suggested fix

A per-step timeout, or a no-output watchdog on the step's stdout, converting an indefinite stall into a recorded failure. P3 steps legitimately take 250–800s on this fleet (18 audits, observed range on carry), so any threshold needs headroom well above that — and the audits' duration varies with corpus size, which differs ~10x between hosts.

Environment

emmett (8 cores), just series --with-model with PROFILES covering 4 live profiles / 6122 transcripts. Host did not reboot (/proc/uptime 121651s afterwards) — it lost only its ZeroTier path, filed separately. carry's concurrent run of the same series completed all 18 audits normally.

## What On emmett, 2026-09-12, the `inference-arbitrage` offload-audit in P3 hung when the host lost its network path mid-API-call. Observed state after ~4.5 hours: ``` PID PPID ETIMES STAT %CPU ARGS 701304 701297 17891 S 0.0 xonsh .../series.xsh --with-model 793226 701304 16340 Sl 0.0 just offload-audit inference-arbitrage 793228 793226 16340 Sl 0.0 just skill inference-arbitrage:offload-audit 793234 793228 16340 Sl 0.1 claude -p --model sonnet ... offload-audit pid=793234 etimes=16340 stat=Sl cputime=00:00:26 wchan=do_epoll_wait open tcp conns: 4 ``` 26 seconds of CPU over 4h32m, blocked in `epoll_wait` on four open-but-dead sockets. ## Why it matters The stall is **indistinguishable from a healthy long-running run**, and specifically invisible where the summary tells readers to look: - the series process stayed alive, so a liveness check passes; - the log did not grow for 4.5 hours, but nothing reads log mtime; - **no rc line was written at all** — an unfinished step produces no record, so it cannot appear in the Not ok list, and the run's own accounting shows 1 of 18 audits with no failures; - meanwhile 16 further audits never started. My own monitor only caught it because I was separately watching the host's reachability, not because anything in the run reported a problem. ## Recovery is trivial once noticed `kill` on the child let the series record ``` [p3] offload-audit: rc=143 FAILED 16369.01s ``` and continue to the next target immediately. So the fix is detection, not recovery. ## Suggested fix A per-step timeout, or a no-output watchdog on the step's stdout, converting an indefinite stall into a recorded failure. P3 steps legitimately take 250–800s on this fleet (18 audits, observed range on carry), so any threshold needs headroom well above that — and the audits' duration varies with corpus size, which differs ~10x between hosts. ## Environment emmett (8 cores), `just series --with-model` with `PROFILES` covering 4 live profiles / 6122 transcripts. Host did not reboot (`/proc/uptime` 121651s afterwards) — it lost only its ZeroTier path, filed separately. carry's concurrent run of the same series completed all 18 audits normally.

oleks commented

2026-09-12 19:59:52 +03:00

Correction to this issue's Environment note

The note says the host "did not reboot (/proc/uptime 121651s afterwards) — it lost only its ZeroTier path, filed separately". That is wrong. emmett hibernated at ~15:40 EEST and resumed at 19:56:15:

Sep 12 19:56:15 emmett systemd-logind[1023]: Operation 'hibernate' finished.

corroborated by the single 255.8-minute gap in emmett's own local load-collector samples, ending at the same second as the resume. /proc/uptime carries across a hibernate/resume within one boot, so it never supported the claim I made from it. The ZeroTier issue I filed alongside this one (oleks/emmett#705) is retracted and closed.

This does not weaken the defect here — it sharpens it.

The trigger is not an exotic network partition. It is the host going to sleep, which is ordinary and will recur. The failure mode is unchanged and now has a mundane cause:

the API call's sockets die across the hibernate;
claude -p never notices, sitting in do_epoll_wait (26s CPU over 4h32m);
the series writes no rc line, so the stall is absent from the Not ok list;
16 further audits never start.

So any overnight or unattended run on a machine that sleeps is exposed, not just one that loses its mesh. That raises the priority of a no-output watchdog over a fixed per-step timeout: a watchdog on stdout would catch a sleeping host, a resumed-but-broken connection, and a genuinely wedged model call alike, without needing a threshold tuned to the 250–800s legitimate range.

Worth noting the run did resume cleanly once the dead child was killed — it recorded rc=143 FAILED 16369.01s and moved to the next target. Nothing about the post-resume state was broken except that one call.

## Correction to this issue's Environment note The note says the host "did not reboot (`/proc/uptime` 121651s afterwards) — it lost only its ZeroTier path, filed separately". **That is wrong.** emmett **hibernated** at ~15:40 EEST and resumed at 19:56:15: ``` Sep 12 19:56:15 emmett systemd-logind[1023]: Operation 'hibernate' finished. ``` corroborated by the single 255.8-minute gap in emmett's own local load-collector samples, ending at the same second as the resume. `/proc/uptime` carries across a hibernate/resume within one boot, so it never supported the claim I made from it. The ZeroTier issue I filed alongside this one (oleks/emmett#705) is retracted and closed. **This does not weaken the defect here — it sharpens it.** The trigger is not an exotic network partition. It is the host going to sleep, which is ordinary and will recur. The failure mode is unchanged and now has a mundane cause: - the API call's sockets die across the hibernate; - `claude -p` never notices, sitting in `do_epoll_wait` (26s CPU over 4h32m); - the series writes no rc line, so the stall is absent from the Not ok list; - 16 further audits never start. So any overnight or unattended run on a machine that sleeps is exposed, not just one that loses its mesh. That raises the priority of a no-output watchdog over a fixed per-step timeout: a watchdog on stdout would catch a sleeping host, a resumed-but-broken connection, and a genuinely wedged model call alike, without needing a threshold tuned to the 250–800s legitimate range. Worth noting the run **did** resume cleanly once the dead child was killed — it recorded `rc=143 FAILED 16369.01s` and moved to the next target. Nothing about the post-resume state was broken except that one call.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: oleks/analytics-playbook#8