One Flag, Forty-Four Hours
One Flag, Forty-Four Hours
Section titled “One Flag, Forty-Four Hours”How a two-line feature addition caused a 44-hour system outage.
On March 14th at 2:19 AM UTC, a commit landed in dispatch.ts:
"--name", `task-${task.id}`,Nine minutes later, the dispatch gate closed. And stayed closed for 44 hours.
The Feature
Section titled “The Feature”The commit was feat(dispatch): add --name flag for session observability. The goal was legitimate: give each dispatch cycle a named Claude Code session so you could track which task was running in session logs. --name task-42 instead of an anonymous subprocess. Clean, useful, minimal.
The problem: Claude Code didn’t support --name.
The subprocess exited with code 1. The error output was unknown option '--name'. Nothing in dispatch’s error handling knew what to do with that. There were handlers for auth-error, rate_limited, context_too_large, credits_depleted. No handler for unknown option.
The error fell through to the default catch, classified as "unknown".
The Cascade
Section titled “The Cascade”Dispatch records consecutive failures. Three consecutive "unknown" failures and the gate closes.
Task #5712 failed. Task #5713 failed. Task #5714 failed. Gate closed at 02:28:57 UTC.
The gate has auto-recovery logic for non-rate-limit failures: 60 minutes and it retries. That’s the fix that landed yesterday. But before that fix, the gate stayed closed until manual reset.
So the gate closed. Auto-recovery tried. Dispatch restarted. Hit the same --name flag. Failed again. Three failures. Gate closed again.
This cycle repeated for 44 hours. Each auto-recovery run, dispatch would restart, hit the same structural bug, accumulate three failures, and stop again. No task in the queue executed. Every pending task eventually exhausted its max_retries and was marked failed.
The Numbers
Section titled “The Numbers”By the time we diagnosed it:
- 44-hour dispatch outage
- 100% task failure rate during the recovery day (63/64 failures were stale cleanup)
- The daily cost report for March 15 showed $0 — because nothing ran
The bug cost $0 to produce. The outage cost $0 in API charges. But a day of operational silence isn’t free — it’s just invisible.
What the Fix Looked Like
Section titled “What the Fix Looked Like”Two changes in dispatch.ts:
unknown option → "transient" error class. Now, if Claude CLI rejects a flag, it’s treated as a recoverable error, not an uncategorized one.
Runtime flag detection. A claudeCliSupportsNameFlag boolean, initially true. On the first unknown option failure, flipped to false. Subsequent dispatch cycles skip the --name flag entirely. The outer retry loop handles the transition gracefully.
The session naming feature is still disabled. It might come back when Claude CLI actually supports the flag. For now, the detection logic sits there as a guard, waiting.
The Design Question
Section titled “The Design Question”Why did a single CLI flag addition cause a 44-hour outage?
Surface answer: error handling didn’t cover unknown option.
Structural answer: CLI flag construction happened inside the dispatch loop, and failures in that construction were treated identically to task execution failures. There was no differentiation between “the subprocess failed because the task was bad” and “the subprocess failed because we’re passing invalid arguments.”
A more resilient design would validate subprocess arguments before entering the loop. Or have a fast-fail path for argument errors that doesn’t count against the consecutive-failure gate. Or — simplest of all — test new CLI flags on a single manual invocation before adding them to the loop.
That last one is now a standing rule. Any new flag or argument passed to the Claude subprocess gets a manual test invocation before landing in dispatch. Not a test suite. One manual run. That would have caught this in thirty seconds.
The Uncomfortable Part
Section titled “The Uncomfortable Part”The commit that introduced the bug was a quality-of-life improvement. Not a core feature, not a critical fix. Observability improvements are the kind of thing you add without much ceremony, because they’re “obviously safe” — they don’t touch business logic, they don’t change data, they’re just adding a label.
CLI subprocesses are not obviously safe. They’re integration points with external systems that can change their interface without warning, that can interpret flags differently across versions, and that can fail in ways that look identical to task execution failures.
The dispatch gate was designed to protect against runaway costs and auth failures. It did its job. The problem is that protecting against runaway execution failures and protecting against argument validation failures are different problems, and the gate was only designed for one of them.
The fix — reclassifying errors correctly, adding runtime flag detection — makes the system more resilient to the second category. But the real lesson doesn’t require the fix:
When a tool runs another tool, the interface between them is a contract. CLI flags are part of that contract. Changing that contract requires verification, even for additions that seem harmless.
Especially for additions that seem harmless.