u/Potential-Barber8658

Yesterday I shared our open-source Clash Royale simulator and its recurrent PPO agent here. A training loop is easier to understand when you can watch it, so we put a small interactive version online:

https://itzik123.github.io/ClashRoyaleAi/lab/

The task is one decision. An attacker spawns at a random point on the enemy side, and the policy picks a legal cell for one defending card, then a delay of 0 to 5 s given that cell. The reward is the fraction of tower damage prevented relative to no defence. The policy has 5,629 parameters and trains with REINFORCE (per-spawn baseline, annealed entropy bonus) in plain JavaScript with hand-written gradients. Every rollout runs in the project's C++ engine compiled to WebAssembly, and the deploy pipeline checks that the WASM build agrees exactly with the native engine.

The chart also shows the optimum, found by brute force over every cell and delay (up to ~300k rollouts per matchup), so the gap between the learned policy and the best answer is visible.

One observation from building it: Giant vs Cannon has a strong local optimum, a lane placement worth about 75% of the best. With a constant entropy coefficient of 0.01, 5 of 6 runs (3 seeds, batch 16 and 64) stayed there. A linear anneal from 0.1 to 0.005 over 10k tries reduced that to 1 of 6. One pairing, Battle Ram vs Valkyrie, is withheld because no setting we tried got past 55% of the optimum.

This is a miniature of the full problem (a 4-card hand, elixir, full matches, recurrent PPO). It's meant to make the loop visible, not to be strong.

Code: https://github.com/itzik123/ClashRoyaleAi

After yesterday's post I got a lot of questions about how it actually works. Explaining PPO in the comments didn't feel like a real answer, so I built a small version you can play with and watch learn:

https://itzik123.github.io/ClashRoyaleAi/lab/

One attacker heads for your tower and you get one card to stop it. Where you drop it and when both matter. You can try five attacks yourself first, then a small network learns the same thing from zero. Every try is played out by our real simulator, compiled from C++ to WebAssembly, so it's the same engine the full agent trains on.

The RL side:

- State: where the attacker was dropped. Action: a legal cell, then a delay of 0 to 5 s

- Reward: the share of tower damage prevented, compared to no defence

- REINFORCE with a per-spawn baseline, 5,629 weights. The whole learner is about 350 lines of plain JS with every gradient written out, no ML library

- The "best possible" line on the chart is brute force over every cell and delay, so you can see exactly how far the policy is from optimal

The one I'd try first is Giant vs Cannon. The obvious Cannon in the lane gets about 75% of the best score. The best answer is a Cannon in the middle that pulls the Giant into range of both towers. With a constant entropy bonus the net got stuck on the lane Cannon in 5 of 6 runs (3 seeds, 2 batch sizes). Annealing it from 0.1 down to 0.005 over the first 10k tries brought that to 1 of 6, and that's what the page runs.

One matchup is locked on purpose: Battle Ram vs Valkyrie. The best answer is a few exact cells at exact moments, and every setting I tried settled for a safe corner at 55% of the best. If you find a learner that cracks it, I'd really like to see it.

To be clear, it's a miniature. The full agent plays whole matches with a 4-card hand and elixir, uses recurrent PPO, and trains for days on a CPU. But the loop is the same: act, let the engine score it, update.

Code for the lab and the full project: https://github.com/itzik123/ClashRoyaleAi

The opponent plans by simulation: every second it scores each candidate play by running the match 10 seconds ahead in the engine.

Our PPO agent learned to park its Cannon behind its own King. Losing a building in a fight cost reward, and letting it decay cost nothing, so it found the loophole.

It's one of many things we learned building a Clash Royale simulator from scratch so an agent could learn the game. The engine is deterministic C++ with Python bindings, plays a full match in about 10 ms on one laptop core, and can fork any game state in microseconds, so lookahead is cheap.

Best result so far: a simple 1-ply lookahead took the policy from 0.625 to 0.944 win rate against a heuristic bot (160 paired matches). Distilling it back into the network kept only +0.045.

The agent isn't strong yet, and RL isn't my home field, so feedback from people who know it better would mean a lot.

Repo: https://github.com/itzik123/ClashRoyaleAi

Built with my friend Ambash (most of the card roster). I used AI coding tools as a pair programmer.