Screeps Reinforcement Learning
I haven't been playing the game for awhile, doing other programming things like machine learning instead. I've learned a lot, especially from Kaggle challenges, and decided to try my hand at a machine learning in Screeps.
Code link: https://github.com/CarsonBurke/xxscreeps/tree/main/samples/rl
First, I pretrain the model on my bot, behavioural cloning so it just tries to copy what the bot does using a bunch of data I collected running it. It can perform all kinds of actions but isn't very efficient or directional.
Then I do reinforcement learning on it. I reward it for harvesting and upgrading and expect the rest to be emergent, which sort of happens. It gets really good at harvesting and upgrading, does stationary harvesters and hauling really well, but also learns to stop building eventually. The rollouts are too short that it doesn't end up seeing the benefits of building. Regardless, I'm happy with the results given my budget.
To get more technical, it's a ~1.5M parameter model with a ViT over patches room tiles (terrain, sources, controller) and entity transformer for the entities (creeps, spawns, towers) each getting a head that outputs actions for the entity. Inspired somewhat by AlphaStar, though I figured autoregressive actioning and temporal recursion would be too expensive for marginal benefit, so I skipped them.
Then a separate 1.5M parameter model critic where I take a VAPO style approach of making the critic Monte-Carlo so it can get signal from the whole rollout for itself and also the actor's advantages. It hurts learning speed a bit, but makes it so the model can learn from very far out returns.
xxscreeps allows me to do really fast parallel rollouts. I think I did 12 games at once for 512 steps, each taking under a couple seconds. If I used the normal screeps engine it would have taken days to train. This partly avoids the immense parallelism that Ben had to do in his approach.
There were a lot of compromises to get it training fast on my single RTX 5090. If I had a bigger budget I would have done longer rollouts to allow it to build and explore more game features. It should be able to claim, colonize and expand too. Maybe a project for someone with more resources than me.
Thanks to Ben for the inspiration!