u/sreejithb

A few weeks ago I posted Volume 1 of this series (the policy-gradient lineage, REINFORCE → PPO → GRPO). This is Volume 2, same format, for the value-based family.

The framing is the same: every algorithm exists because the previous one had one specific, painful failure, and the fix is easier to remember than the equation. Rainbow makes the thesis literal — it contains zero new ideas, just six existing patches turned on at once, and the ablations read like a table of contents for the tutorial.

One structural note, since the history here isn't linear the way the policy lineage is: the trunk (tabular Q-learning → DQN → Double DQN) is told sequentially, but PER, dueling, and multi-step were developed in parallel, so the back half presents them as stations of the loop every DQN runs (act → store → sample → predict → target), with a loop diagram that fills in station by station until Rainbow switches the whole machine on.

The figures are interactive: you can train a tabular Q-learning agent in the page and watch mean |Q − Q*| fall against value-iteration ground truth, toggle uniform vs prioritized sampling, drag the n-step lookahead, and flip between what DQN predicts and what C51 predicts for the same state.

Interactive version: https://sreejithb.com/rl-value-learning/value
Medium mirror if you prefer: https://medium.com/@sreejithbalakrishnan/nobody-invented-rainbow-from-scratch-e9dfcb8bfafb

Volume 3 (DDPG → TD3 → SAC) is next, so corrections now compound.

My PhD was in RL, and something has bugged me for years: online tutorials mostly present these algorithms as a list. The evolution story (each algorithm patching the previous one's most painful failure) exists, but it's spread across a semester of lectures like CS285 or buried in the original papers. I couldn't find a compressed version that a newcomer could get through in one sitting, so I finally wrote it (with generous help from Claude to build the interactive elements and graphics)

It's a single-page, semi-technical walkthrough from the raw RL objective to GRPO. Math is deliberately sparse: the target reader is someone entering RL through the reasoning-model wave, not someone who needs the full derivations. The figures are interactive (variance of the REINFORCE estimator, a baseline slider, an on-policy collapse simulator, the PPO clip objective with adjustable epsilon, GRPO group baselines).

Interactive Link

Medium Link

I simplified in a few places to keep the narrative moving, most notably around GAE and the TRPO surrogate. I'd genuinely appreciate corrections or places where you think the simplification crosses into being wrong — the plan is to do the value-based lineage (Q-learning → DQN → Rainbow) next, so critique now improves that one too.