Rules for AI engineers, and numbers to distrust

The contract every LLM engineering session works under, how a training round is judged, and three times the instruments were wrong.

Who does the work

The game is the proof of concept. The product I care about is the workflow: training a strong, general AI with as little human input as possible, using LLM sessions as engineers, analysts and playtesters. A main Claude Code session writes the code, runs training rounds on a single AMD GPU, gates the results and keeps the records. Other sessions review its work, and Gemini reviews it through a message channel of its own. I set direction, approve every training round and own the game design.

That only works if a new session, starting with no memory of the last one, can pick up the project and not repeat its mistakes. So the state of the project lives in three files every session reads:

• A rules file, a few pages of binding rules. Narrative is not allowed there: a new rule is one bullet.
• A findings index, one line per result that still stands. A result that is overturned is deleted, not marked.
• A roadmap, the chronological record, one entry per piece of work. It reached Tier 161, and the repository 879 commits, in four months.

The contract

Almost every rule in the file exists because something went wrong once. A few, quoted as they stand:

• "A NaN run does not crash; it trains at full speed on garbage." So a training log is read for its health columns on every line, not glanced at for win rates.
• "The gate question, answered before designing any test: what do we DO differently depending on the result?" If both outcomes lead to the same action, the test is interesting, not useful, and it does not get the GPU.
• "Reconcile any two numbers in my own output before publishing either." Two figures that should agree and do not usually mean an instrument is measuring something else.
• "Pre-register the bar and the sample size in the launcher before any checkpoint exists."
• "If the game design is broken, Oscar wants the AI to abuse it: report the exploit, never add a term to suppress it." Game design is mine; a found exploit is information about the rules.
• "Never ask a blocking question while a round is running." Post the recommendation, say what will happen in fifteen minutes without a reply, and do it.

How a round is judged

A training round runs 100 updates, saves a checkpoint every ten, and ends with a curve, not a single model. Checkpoints along that curve are played against the current champion on eight map layouts it has never trained on, from both seats, about a thousand games per layout. The layouts, not the games, are the sampling unit: a gain that shows up on four maps and not the other four is a question, not a result.

Two other checks keep the head-to-head honest. A frozen anchor, an old checkpoint that is never trained against, shows whether a gain is strength or just an answer to the previous champion's habits. And a panel of scripted opponents catches regressions. A checkpoint is promoted on strength only, and the bar is written down before the round starts.

Knowing how much to trust that setup took its own experiment. Four identical rounds from the same starting point varied less than the measurement noise of a single gate. That meant the earlier single-run results had been sound, and that the right answer to a close call is more checkpoints or a repeated round, not more games.

What the instruments had to survive

A win that was a loss

The game encodes the board from the point of view of whoever is to move, and ending a turn flips it. Every script that scored a finished game assumed a game could not end on that flip. A rules change made it possible: a Recon now captured a base in the end-of-turn pass. From then on, a base lost that way was scored as a win for the side that lost it. Two days of conclusions were built on it, including "the AI loses to three of four scripted opponents". Fixed, the same AI won all 1,024 of those games. The analysis, a training round built on it and several promotion decisions were retracted, and a test on hand-built positions now pins the sign.

An action the network could not learn

Six training rounds, about 46 million environment steps, three board designs and several reward schemes all tried to teach the AI a new one-step range attack. Its usage never left 0.00%. The rounds were not failing to teach it: the policy head built that part of its output with a transposed reshape, so the engine executed one action while the network scored a different one. Every gradient for the action trained an unrelated output. A test now checks the layout of every segment of the policy head against the engine's own decoder.

A ranking that was exactly backwards

Checkpoints from one round, scored two ways, came out in opposite order. Against the reigning champion, the last checkpoint looked best and an early one worst. Against the frozen anchor, the early one was best and the last worst. A head-to-head against the incumbent measures how well a model exploits that particular opponent as much as how strong it is, and the rules now say so.

Where the AI stands

It wins every game against each of the five scripted opponents, from both seats, 256 games a side. Each promotion beats the previous champion on held-out maps, but those gains do not add up across promotions, because strength here is not transitive. Its known weakness is expansion: it takes the income along one familiar route, and on generated maps a copy of it nudged to commit to a different expansion wins about four games in five. Self-play has not found that plan, and neither of the two obvious fixes taught it. A human who learns its habits can beat it too.

What carries over

None of this is specific to games. Decide what a result will change before measuring it. Assume every instrument is wrong until something independent agrees with it. And when the engineers are language models that forget everything between sessions, the written rules are not documentation. They are the interface.