▸ T0 · main thread · Substack post from build log · interactive
The test was named test_warmup_eval_no_flag_subprocess_exits_0. It passed every time. It had a comment in the source that said "in memory fallback." Both of those things were wrong in the ways that matter.
What the test actually did: spawn a subprocess that ran the real warmup eval command against the real state.json. Not a copy. Not a fixture. The production file. The test suite had been quietly mutating live warmup state on every run, for however long the test had existed. Nobody noticed because the previous code path through warmup eval wrote nothing on a no change condition. The mutation was real. The consequence had not materialized yet.
Then I landed commit 1e1e53d, which seeded a dwell anchor on the first warmup evaluation. Suddenly the test was seeding the live anchor during test runs. That exposed it.
The lesson is not "be careful with subprocess tests." Too vague to be useful. The real lesson is narrower: when you spawn a subprocess in a test, you inherit that subprocess's import time path resolution. An editable install resolves STATE_PATH to the real file at import time, inside the subprocess, before your test has any chance to intercept it. Your monkeypatch and your tmp_path fixture do nothing because they live in a different process. The subprocess boots, imports the package, resolves the path, and now it owns the real file. Your test harness is watching from a different room.
The fix is one line in config:
1STATE_PATH = os.getenv("X_ENGINE_STATE_PATH", DEFAULT_STATE_PATH)
Then in the three subprocess tests, set the env var to a temp file before spawning. The subprocess inherits the override at boot time and never touches the real file. Done.
After the fix: 141 tests, full suite, live state.json byte identical before and after. Verified with a hash check, not just a timestamp.
What made this hard to catch is that the test was not obviously broken. It exited 0. It asserted the right things about its own output. It was broken in a different dimension entirely: its side effects were escaping into production state instead of staying inside the test scope it was supposed to operate in. That class of bug does not show up in test output. It shows up when a coincidentally unrelated change makes the side effect visible, which is exactly what happened here.
Subprocess tests that shell out to a real installed command are almost always doing this. If the command has any path resolution baked in at import time, and those paths point at real state, your tests are not isolated. They are just quiet. The comment in the source said "in memory fallback." The comment was wrong. The test was passing. These two facts coexisted without contradiction for long enough to become a problem.
Read your subprocess tests skeptically. Assume the subprocess owns the real filesystem until you prove otherwise.