I built a reinforcement learning environment around Pokelike! Try to beat it!
Hey everyone!
I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while. A few days ago my friends showed me Pokelike, a small Pokémon roguelike that runs in the browser. The first thing I thought was that it could be pretty fun to turn it into an RL environment.
So I did.
The repo is here:
https://github.com/pierpierpy/pokelike.xyz.bot
The idea is to run the actual game locally and expose its state and actions to an agent. There is no image processing involved. The bot gets the game state directly and decides what to do next. The agent has to make the decisions around the battles, like where to go on the map, which Pokémon to catch, which items to take, when to swap Pokémon and which moves to learn.
One thing I found interesting is that the map forces you to make some decisions quite early. Once you choose a node, the other nodes on that layer are gone, so deciding where to go can matter quite a bit later in the run. There is also a lot of information in the state that could potentially be useful, but I'm still not sure what the best way to represent it is.
I've already implemented a few basic RL agents. There is currently a Dyna-Q agent and two linear SARSA agents in the repo. The results are not amazing yet. On the current benchmark, random gets around 0.56 badges, Dyna-Q gets around 0.62, while the two SARSA agents get around 1.30 and 1.36. The two SARSA agents mainly differ in their state representation, with the better one using 100 hand-designed features instead of 81.
This is probably the part I'm most interested in at the moment. Finding a good state representation seems to make a pretty big difference, and the environment has some properties that make it a bit more interesting than I initially expected. The reward is fairly sparse, the action space depends on the current state, and some decisions only become useful several steps later.
At the same time, the environment is completely reproducible. With the same seed and the same actions you get exactly the same run. For the current leaderboard I'm using 50 fixed seeds, so different agents can be compared on the same games.
The interface is also intentionally pretty simple. You can basically implement a bot that takes the current state and returns an action. It doesn't have to be a specific RL algorithm either. You could try DQN, PPO, search, a hand-written policy, or pretty much anything else.
I'm still experimenting with the environment and the agents, so I'd be really curious to see what other people would try. In particular I'd love to see if someone can get significantly better results with a better state representation or a different reward function.
you can experiment with the environment with no efforts, just follow the guide and readme, setup the enrinvonment, experiment a bit and then if you like the result, you can create a pull request to the repo with your bot in the bots/ folder (everything is clearly explained in the GUIDE.md)
If you want to try it, everything is in the repo
https://github.com/pierpierpy/pokelike.xyz.bot
If you find bugs or have ideas for the environment, please let me know. I'm happy to make changes if there are things that would make it more useful for experimenting with RL.
The whole thing also runs offline. During setup it downloads the game and the required assets, and after that everything runs locally.
I originally started this because I thought it would be a fun little RL project, so I'm mostly curious to see how far people can push it.
