Runs in your browser, on a computer or a phone.
This project started as two questions:
To fully explore these two questions, I needed the source code of a game. So naturally I developed my own game: Robostrat. Robostrat is a turn-based strategy game played on a small hex-tiled board for two players. It is designed to have a rule set simple enough that it is easy for a human to learn, but complex enough that a bot would need to be more than just a tree-searching script over the available moves. The ultimate goal is to develop a workflow for AI training that is transferable to other turn-based strategy games.
Since my private data center consists of a sole Radeon RX 7900 XT, a pure self-play approach with no guidance (i.e. throw compute at it until it git gud) was not a reliable strategy to train a bot. Instead I needed a more guided approach, where specific weaknesses of the bot are identified by an outside observer and training is allocated to try to fix these weaknesses. This, together with a very exploratory approach to various bot designs, has been the main focus of this project.
At its core, the bot is a single neural network. It reads the board as a few dozen numbers per hex (terrain, owner, unit type, health, supply and so on) and answers with two things: how likely it is to play each possible action, and how likely it thinks it is to win. The network is a small convolutional network, which looks at every hex the same way and so is not tied to one map. Its main body comes in four versions, one for each stage of the game (turns 1-10, 11-30, 31-60 and 61 on), and the turn number decides which one runs, so about a million of its 4.2 million parameters are used for any one action. A turn in Robostrat is a sequence of single actions (move this unit, attack with that one, buy a unit, end the turn), so the network picks one action at a time.
It is trained with PPO, a standard reinforcement learning algorithm, by playing about a thousand games at a time against itself, a pool of other versions and a scripted opponent. The network trains on the GPU, while the rules run on the CPU in the game engine, written in Rust. The game on this site runs the same engine, compiled to WebAssembly, so the bot plays here by exactly the rules it was trained on. Training runs in rounds of about two hours on a mix of maps, mostly a fresh generated map every game. A new version only replaces the current best after being tested against it on eight maps that neither of them has trained on, and against a third version that neither of them has trained against, as an independent check.
The bot you play on this site is one of those champions, exported to run in your browser. By default it also thinks ahead: at the start of each turn it tries 16 different ways to begin the turn, plays each one out to the end of the turn, tries a few likely replies from its opponent, and plays the turn its network rates best. That takes a few seconds per turn, against about 16 milliseconds per action for the network alone.
The current status of the project is that I have managed to train a bot that is somewhat competent on a variety of maps, in the sense that it sometimes wins against me. Once I learn the quirks from the last session of training, however, the bot usually fails to score any new victories against me. Regarding game design, the training process has helped me to identify and fix some flaws, but the workflow is still some way from the autonomous game balancing loop I had hoped for.
Published on this website is a version of the game, which can be played in a browser against one of the bots I trained, or watched while the bot plays against itself. Either side can also be played by a language model, which reads the same board text as the scouts in the devlog below. I also publish a devlog here, which dives deeper into some of the experiments done and challenges encountered in this project.
How the game is turned into text a language model can play, and which models turned out to be good at it.