Letting language models play my AI

An MCP server that turns a strategy game into text, what the LLM scouts found that no test did, and what they turned out not to be good for.

The blind spot of self-play

The Robostrat AI learns by playing against copies of itself. That is a very efficient way to get good, and it has a built-in blind spot: a plan that neither copy ever tries is never punished, so the AI never learns to answer it.

For a while the fix was me. Every strong scripted opponent I wrote came from a person noticing something the AI could not handle, usually me. That does not scale, and it depends on me having the idea. What I wanted was an automatic version of that playtester: something that plays differently from the AI, adapts during a game, and writes down what it tried.

The game as text

Language models are poor at reading a tensor and good at reading a board description, so the scout gets its own interface: an MCP server wrapped around the fast Python copy of the game engine. Any MCP client can drive it. In practice that has been Claude Code, Gemini through Google's Antigravity, and local Qwen and Gemma models through a small driver script.

Each turn the model gets a text board: the map, its units and the enemy's, income, and a layer of derived relations such as which enemy can reach which tile, which units are cut off from supply and which tiles an overwatch covers. It answers with short commands:

move U1 to (-3,0)       attack U2 -> E1        overwatch U2 facing (1,0)
buy Recon at base       range U3 -> E1         end turn

and a handful of tools around them:

Three design choices that mattered

The board cannot lie about the rules. Every action the render lists as available is derived from the engine's own action mask, so a scout never plans around a move that does not exist. Prose is harder to keep honest, so every sentence in the render that asserts a rule is registered against the test that proves it. When a scout quoted a repair price range that does not exist, the cause was that the render had never printed the price. The fix was to print it.

The simulator never consults the AI. simulate lets a scout look ahead, but the opponent's moves in a hypothetical are the scout's own guesses. If it stepped the real network, every win would be search against an oracle and worth nothing, so a test checks exactly that, including on the machine that runs the scout games.

Knowledge outlives the game. Each scout keeps a playbook it can rewrite and a log it can only append to. The engineering session can pin facts above the rewritable part. One such note, written after a scout's playbook claimed an exploit its own game records contradicted, reads in part:

Every game ended the same way: you finished with 0 of 5 units supplied while the champion had 6-8, and "no supplied units + enemy holds your base" IS the loss condition.

What the scouts found

The most useful scout game was one that did not win. A Claude session playing the weaker seat on a map the AI had trained on reached turn 31 roughly level, won nearly every fight and still lost the economy. Along the way it noticed that an Artillery's range attack draws no counter-attack and no overwatch reaction, and that the AI almost never used it. Measured afterwards over self-play, the attack was available in about a quarter of the AI's turns and used in 4.5% of them. That became a line of training work of its own.

The rest were things no test had seen, found by playing whole games and by trying to replay them afterwards:

• A scripted opponent broke ties between equally good moves with the process-wide random number stream, so replays of scout games silently diverged, dozens of moves before any error appeared.
• Game logs named the opponent by a file path that every promotion overwrites, so each new champion quietly invalidated every earlier game. Logs now pin the opponent by hash.
• The AI's indirect fire was missing from the move narration the scouts read.
• An income bug involving disconnected cities.
• The AI had memorised one expansion route rather than the principle behind it, which later became one of the project's main open problems.

Does thinking ahead matter? An experiment

Gemini, playing from its own written doctrine, won three out of three games on one generated map. Two local models lost to the AI in the same narrow band, around turns 12 to 20, and the largest behavioural difference was how much they used simulate. So we took the tool away from the strong player: same map, same seat, same three seeds, simulator disabled and verified unused in all three logs.

Without its lookahead, the strong model landed exactly in the weak models' band. The prediction was registered before the games were played. One confound is on the record too: the prompt told Gemini a loss was an acceptable result, which may have lowered its effort.

What scouts are not good for

Explaining the opponent. Scouts are reliable about what they did and inventive about why the AI did what it did. One report described a blunder the AI made in "3 / 3 opportunities". The logs showed one of three, and the mechanism it described appeared in none of them. The prompt had asked for counts rather than estimates; checking against the logs is what caught it.

Training data. A scout plays a few games an hour; the trainer plays thousands of game steps a second. Starting training games from positions the scouts reached taught the AI those positions and nothing that carried over to new maps.

Lasting evidence. A position that was hard for last month's AI is not necessarily hard for this month's, and a rules change can make an old transcript unplayable. Scout games have a shelf life, and the logs now record enough to tell when one has expired.

What a scout is

A detector. It is a cheap, articulate, adversarial playtester that plays whole games against the current AI, writes down what it tried, and turns up the bugs that only appear when a game is played and replayed end to end. The decisions about what to do with what it finds stay with the measurements.