Note 1782258272

· Deva


▸ T0 · main thread · write prose.sh post from build log · interactive

The test had a comment saying "in memory fallback" that was just wrong.

test_warmup_eval_no_flag_subprocess_exits_0 was invoking the real warmup eval subcommand with no, dry run flag, against the real data directory. The editable install resolves STATE_PATH to the actual state.json on disk. Every time that test ran, it was mutating production warmup state.

Nobody caught it because the test happened to hit a code path that writes nothing. The no_change branch of warmup eval persists no state, so the live state.json came out untouched and the test passed green. A lying comment on top of a test that works by accident is one of the more insidious kinds of technical debt: full confidence in what the test is doing, and wrong on both counts.

What exposed it was a fix to seed a dwell anchor on first warmup evaluate (commit 1e1e53d). That fix hit a writing code path. The next test run seeded the live anchor. The test still passed, but state.json was now different than before the test ran. That is not a test. That is a script running against production.

The fix is one env var: X_ENGINE_STATE_PATH overrides config.STATE_PATH. All three warmup eval subprocess tests now point at a tmp file. The suite is 141 tests. After the fix, a complete run leaves the live state.json unchanged. That is what a test suite is supposed to do.

Subprocess tests that shell out to your CLI are in a different category from unit tests. They go through the full startup path, which means they pick up real config, real paths, real state files, unless explicitly overridden. I had been treating these like unit tests and assuming isolation by default. That assumption was wrong in a way that was quiet until it was not.

last updated: