The subprocess call with no ceiling: an hour-long hang and the fix that closed it

A daily automation pipeline I run went quiet for an hour. Not crashed. Not erroring. Just... still running, long after it should have finished, with no log line to explain why. That silence turned out to be the whole bug.

What the Pipeline Looked Like

The pipeline runs in stages: scrape a source, qualify what it finds, apply an LLM step to generate content, submit the result. Each stage calls out to something external — a website, an API, a subprocess. Under normal conditions the whole thing finishes in well under an hour.

One stage shells out to a command-line LLM tool as a subprocess and waits for it to return. That call had been reliable for weeks. Reliable enough that nobody had asked the question that actually mattered: what happens if it doesn't return?

The Morning It Didn't Return

The run started on schedule. Every stage up to the LLM call logged normally. Then nothing. No error. No timeout message, because there was no timeout to trigger one. No crash, because a hung subprocess isn't a crash — it's a process sitting there, alive, waiting on something that was never going to finish on its own that day.

An hour later, the run was still "in progress," according to the only signal available: the process hadn't exited. Everything downstream of that stage — the rest of that day's scheduled work — was blocked behind a single subprocess call that had no way to give up.

Why It Took an Hour to Even Notice

This is the part worth sitting with. A hang doesn't announce itself. A crash gives you a stack trace and a timestamp. A hang gives you silence, and silence looks identical to "still working" right up until someone checks the clock and realizes normal runs finish forty minutes faster than this.

The subprocess spawn had been written the way most subprocess calls get written the first time: call it, wait for it, use the result. That's correct for the common case. It has exactly one blind spot — it assumes the external process will always return in a reasonable window, and "reasonable" was never actually enforced anywhere. It was just an assumption baked into the code by omission, the kind you only notice once it's already cost you an hour.

The Fix

Two changes, at two different scopes.

At the level of the individual subprocess spawn, I put an explicit ceiling on it — a hard cap on how long that specific call is allowed to run before it's killed and treated as a failure rather than a pending result. A few minutes was generous enough for any legitimate run and tight enough that a genuine hang gets caught and reported instead of silently eating the rest of the day.

At the level of the pipeline itself, I wrapped every external-facing step — every scrape, every qualify pass, every apply step — in its own timeout, independent of the subprocess-level one. The reasoning: a hang can happen anywhere something reaches outside the process, not just in the one call that happened to hang this time. Fixing only the specific call that bit me once fixes the incident I already had. Wrapping every external call at the pipeline level fixes the next one I haven't had yet.

Proof It Actually Works, Not Just Looks Fixed

I don't trust a fix like this until I've seen it fire for real, not just in a synthetic test where I force a hang on purpose. The first real run after shipping it: two genuine timeouts fired mid-run — real external calls that would have hung the way the original one did. Both got caught, killed, logged, and the pipeline kept moving instead of stalling. The full run finished in a little over half an hour instead of hanging indefinitely.

That's the bar that actually matters. Not "the timeout code exists." Not "it passed a test where I mocked a hang." A real run, real external dependencies, real timeouts firing on their own, and the system recovering instead of going quiet.

The Broader Lesson

Any call to something outside your own process — a subprocess, a network request, a browser automation step — has an implicit timeout by default: infinite. If you haven't set one explicitly, you've set one implicitly, and it's the worst possible value. It doesn't matter that the call has been reliable for weeks. Reliable-so-far and bounded are different properties, and only one of them protects you.

The fix isn't glamorous. It's a handful of timeout wrappers around calls that already existed. But the failure mode it closes — an entire day's automation going silent for an hour with nothing in the logs to explain why — is exactly the kind of bug that's cheap to prevent and expensive to diagnose after the fact. Add the ceiling before you need it, not after the first hour you lose to not having one.