r/reinforcementlearning

My PhD was in RL, and something has bugged me for years: online tutorials mostly present these algorithms as a list. The evolution story (each algorithm patching the previous one's most painful failure) exists, but it's spread across a semester of lectures like CS285 or buried in the original papers. I couldn't find a compressed version that a newcomer could get through in one sitting, so I finally wrote it (with generous help from Claude to build the interactive elements and graphics)

It's a single-page, semi-technical walkthrough from the raw RL objective to GRPO. Math is deliberately sparse: the target reader is someone entering RL through the reasoning-model wave, not someone who needs the full derivations. The figures are interactive (variance of the REINFORCE estimator, a baseline slider, an on-policy collapse simulator, the PPO clip objective with adjustable epsilon, GRPO group baselines).

Interactive Link

Medium Link

I simplified in a few places to keep the narrative moving, most notably around GAE and the TRPO surrogate. I'd genuinely appreciate corrections or places where you think the simplification crosses into being wrong — the plan is to do the value-based lineage (Q-learning → DQN → Rainbow) next, so critique now improves that one too.

I've been trying to get into rl for a long time but I don't see any good resources out there. help me out!

https://preview.redd.it/e0ydm43a55kh1.png?width=2902&format=png&auto=webp&s=8c9b88ff5e4157811e8996ba5a1e96cc55c8ae6a

This paper is using a much narrower definition of recursive self-improvement than the phrase usually suggests.

AQuA stores validated evidence in a persistent research state that shapes later hypotheses. The underlying language model and evaluator remain fixed. I still find the narrower claim interesting, even if it sits closer to memory-augmented research automation than to a model rewriting itself.

The paper does not establish any weight-level capability gain. Is persistent memory that improves later research decisions enough to call a system RSI, or should the term require changes to the system’s underlying capabilities?

I'd like to share reinfors, an open-source RL library that runs the engine in rust while the network and training loop remain ordinary python. Here is a quick overview, with full documentation available in the repo for anyone that's interested.

Motivation

During my own research, I found that existing open-source libraries did not offer the balance between modularity/composability and performance that I wanted. Python-first stacks are highly flexible but make simulation/search the bottleneck. Native frameworks (e.g. all-C++) keep the hot loop fast, but typically pull training into native code with it. Fully fused pipelines are generally the fastest, but their specialisation sacrifices flexibility.

Approach

reinfors is designed so that everyday use requires only python. Composing, training and evaluating never touch lower-level code. You compose an engine from the built-in games and algorithms, pass it an inference callback, and keep the network, optimizer and training loop as ordinary python code (pytorch, JAX, or anything else). Throughput-sensitive parts of the pipeline (simulation, search, episode orchestration, batch assembly) run in rust underneath. The engine pools inference requests across games and search leaves into numpy batches, so the python boundary is crossed rarely.

Does the python boundary cost performance?

This was my primary concern, so I ran some benchmarking experiments. I compared reinfors against an all-C++ libtorch implementation (OpenSpiel) on chess alphazero training, with the network, search budget and gradient intensity held equal. reinfors came out slightly ahead on throughput, and its trained agents performed slightly better in head-to-head games. Note that this was run on a single AWS instance (g5.2xlarge — one A10G GPU, four physical cores), so it is not a general claim, but it is evidence that the boundary need not cost meaningful performance. See link in repo README for details if interested.

What's in it today:

- Games — chess, backgammon, connect4, snake, gridworld, three poker variants

- Algorithms — alphazero, MCTS/treestrap, expectimax, minimax, DQN, PPO, CFR, Deep CFR, MCCFR

Current limitations:

- Fixed discrete action spaces only

- Sequential and simultaneous games only (no mixed-phase games that switch between sequential and simultaneous)

- The game/algorithm catalogue is still small

- New games and algorithms are written against rust traits (python is only for composing and training, not for defining new components)

I hope this may be useful for researchers wanting to experiment with the existing game/algorithm combinations (the examples should get you running quickly). For those comfortable with rust, the codebase is also designed to be extended. New games, policies and learners are written against small composable traits, and the extension guides should provide sufficient details.

Feedback and contributions are very welcome!

Repo: github.com/jeepjeepjeep/reinfors

Hi everyone,

I’m working on a bachelor research project on using reinforcement learning for dynamic obstacle avoidance in the CARLA simulator.

My current setup is:

• CARLA simulator
• LiDAR observations
• Raw 3D point cloud reduced to a 1D array of 360 minimum-distance values
• Stable-Baselines3
• DQN
• Discrete action space with 36 steering/throttle combinations
• Custom Gym environment
• The goal is to avoid dynamic obstacles while continuing to drive forward

I chose DQN partly because my action space is discrete, but I’m interested in whether this was actually a sensible choice for this type of problem.

I’d especially appreciate feedback on these points:

1. DQN vs PPO/SAC

Given the 360-value LiDAR observation and discrete steering/throttle action space, would you consider DQN a reasonable baseline?

Would you expect PPO or SAC to have significant advantages here, and if so, why? I’m particularly interested in whether the continuous-action capabilities of algorithms such as SAC would actually be useful enough to justify changing the action space.

2. Reward design

How important is reward engineering in obstacle-avoidance tasks like this?

My main issue is finding a balance between rewarding forward progress and penalizing collisions/unsafe behaviour without encouraging unwanted behaviour such as simply stopping to avoid obstacles.

Are there particular reward-design principles or common failure modes I should be aware of?

3. CARLA and RL

For research into RL-based autonomous driving, do you consider CARLA a suitable environment, or are there other simulators/frameworks you would recommend?

I liked using CARLA but it was pretty heavy which made it so I didn't run as many training runs

Thanks in advance for any feedback or suggestions!

Not a transformer. It's a recurrent latent reasoning setup that keeps "thinking" in latent space before answering. Sits completely outside the published cost/accuracy frontier for ARC-AGI, and something this size runs on basically anything. Paper is from the Pathway team, dropped 4 days ago. I want to see it scaled to 1-3B before getting too excited, but the shape of the result is wild.

Hey guys,

I think people on this sub are incredibly capable and wise, I am very inspired by your projects. Therefore, I’d appreciate your tips so much.

I study robotics in Germany in masters, had theoretical courses in learning based control and reinforcement learning at a renowned technical university. I apply my theoretical knowledge in hackathons regularly, I also have my own projects with serial kinematics. I had a computer vision internship in robotics industry. I really want to gain industry experience in RL. Especially applied in fields such as construction, mining, where the work has positive contributions and impact on communities. I had completed an excavator policy optimisation project with ROS2 due to my interest. I get the impression that being accepted to a position is impossible and my efforts are not sufficient. What am I missing?

I haven't been playing the game for awhile, doing other programming things like machine learning instead. I've learned a lot, especially from Kaggle challenges, and decided to try my hand at a machine learning in Screeps.

Code link: https://github.com/CarsonBurke/xxscreeps/tree/main/samples/rl

First, I pretrain the model on my bot, behavioural cloning so it just tries to copy what the bot does using a bunch of data I collected running it. It can perform all kinds of actions but isn't very efficient or directional.

Then I do reinforcement learning on it. I reward it for harvesting and upgrading and expect the rest to be emergent, which sort of happens. It gets really good at harvesting and upgrading, does stationary harvesters and hauling really well, but also learns to stop building eventually. The rollouts are too short that it doesn't end up seeing the benefits of building. Regardless, I'm happy with the results given my budget.

To get more technical, it's a ~1.5M parameter model with a ViT over patches room tiles (terrain, sources, controller) and entity transformer for the entities (creeps, spawns, towers) each getting a head that outputs actions for the entity. Inspired somewhat by AlphaStar, though I figured autoregressive actioning and temporal recursion would be too expensive for marginal benefit, so I skipped them.

Then a separate 1.5M parameter model critic where I take a VAPO style approach of making the critic Monte-Carlo so it can get signal from the whole rollout for itself and also the actor's advantages. It hurts learning speed a bit, but makes it so the model can learn from very far out returns.

xxscreeps allows me to do really fast parallel rollouts. I think I did 12 games at once for 512 steps, each taking under a couple seconds. If I used the normal screeps engine it would have taken days to train. This partly avoids the immense parallelism that Ben had to do in his approach.

There were a lot of compromises to get it training fast on my single RTX 5090. If I had a bigger budget I would have done longer rollouts to allow it to build and explore more game features. It should be able to claim, colonize and expand too. Maybe a project for someone with more resources than me.

Thanks to Ben for the inspiration!

Hi all, my studying area is traffic engineering in networks and I'd like to develop my DQN-GCN code with distributional DQN but it seems there are a lot of options such as C51, QR-DQN etc. . Do you have any suggestions to see and understand problem & variation mapping before starting the implementation? Sure, AI tools suggest some but I prefer your advices more than AI tools.

Thanks!

Hi everyone — I'm a complete beginner working on a small project: an agent that decides whether to remind, wait, or notify a caregiver when someone hasn't responded to a medication reminder. The true state (fine but busy, asleep, phone dead, actual emergency) is hidden, and I only get noisy signals like elapsed time and response history.

I've been framing this as a POMDP — belief state over hidden states, updated via Bayesian filtering, with an escalation policy mapping belief → action.

Given the asymmetric costs (missing a real emergency vs. a false alarm), does POMDP even make sense here, or is this overkill for what could be solved with a simpler heuristic/threshold approach? Would appreciate any pointers or papers on similar "when to escalate" problems under uncertainty.

Hello. Is somebody well acquainted with the status of Montezuma's Revenge and Atari games generally in 2026? Most of the information comes from 5 years ago. It seems that in the past there were many doubts about the level of using the domain knowledge, imitation learning etc. Are there any obvious milestones still to be achieved?

Be well!