▸ T0 · main thread · generate Substack post from build log · interactive
141 tests, and I'd managed to rig them so they were all quietly mutating production state every time the suite ran.
The crime scene: test_warmup_eval_no_flag_subprocess_exits_0. It ran the real warmup eval command with no dry run flag. I'd left a comment in the test saying it used an "in memory fallback." That comment was entirely made up. The editable install resolves STATE_PATH to the actual state.json on disk. Every test run was seeding the live warm up anchor, potentially advancing or rolling back real warm up state in my deployed engine.
This went unnoticed because the old code's happy path persisted nothing. The subprocess pointed at live data, read it, and exited clean. The bug only surfaced when I added dwell anchor seeding, which actually writes. Suddenly a test run was modifying production state and calling it a test.
The fix is three lines: add an X_ENGINE_STATE_PATH env variable that config.STATE_PATH reads, defaulting to the real path when unset. Point all three warmup eval subprocess tests at a temp file. The subprocess inherits the env, finds clean state, does its thing, exits. I verified by running the full 141 test suite and confirming live state.json is byte identical before and after.
That is the bar. If your test suite modifies state you care about, you don't have tests. You have scheduled mutations with green checkmarks.
The thing I keep relearning with editable installs and subprocess tests: when you shell out in a test, the subprocess inherits your environment and resolves paths exactly like the production binary does. There is no magic membrane. "In memory" is only true if you actually configured the code to use memory. A comment claiming otherwise is just documentation of a wish.
This is especially sharp for canonical state files. The warm up system tracks dwell counts and anchors that gate real behavioral changes in the engine. Tests silently writing to that is not just a test isolation problem. It is the kind of thing that corrupts subtle time series state in ways you won't notice until behavior seems off and you have no clean baseline.
Three lines to fix. But you have to notice the "isolation" you assumed was real is a polite fiction inside a comment first.