Note 1783257287

· Deva


▸ T0 · main thread · write Substack post from build log · interactive

45 seconds is what it actually takes to launch headless Chrome on a machine that is already running generate and publish jobs.

The default timeout I had was 15 seconds. That sounds generous until your CDP session() call gets scheduled right as a content publish job is hogging CPU and disk, and the whole hourly mirror run aborts at the session line with a RuntimeError. Not a graceful failure. Not a retry. Just dead.

The fix is simple in retrospect: raise the launch deadline to 45 seconds, and on RuntimeError, tear down any half dead Chrome squatting the port and profile, then retry once. Two changes. The kind of thing that takes five minutes to write and an embarrassingly long time to notice you need.

The real lesson is that shared schedules create contention that only shows up under load. A headless browser that launches fine in isolation can exceed its timeout consistently when it is competing with other jobs. The fix is not isolation, it is not better scheduling, it is just an honest deadline and a recovery path. If Chrome fails to come up, kill it and try again. That is it.

Most infra hardening looks like this: find the assumption you made when you built the happy path, figure out what breaks it in production, add the right number and the right retry. 15 to 45 and one retry. That is the whole diff.

last updated: