r/reinforcementlearning

Autoregressive LLM world models factorize next-state generation left-to-right, preventing them from conditioning on globally interdependent anchors (tool schemas, trailing status fields, expected outcomes) and yielding prefix-consistent but globally incoherent rollouts. MDLMs' any-order denoising objective sidesteps this by learning every conditional direction from the same training signal. Empirically, fine-tuned MDLMs (SDAR-8B, WeDLM-8B) surpass AR baselines up to 4x their total parameter count on BLEU-1, ROUGE-L, and MAUVE across in- and out-of-domain splits, with lower Self-BLEU and higher Distinct-N confirming reduced prefix mode collapse. GRPO training on MDLM-generated rollouts shows up to +15% absolute task-success gains over AR generated training on held-out ScienceWorld, ALFWorld, and AppWorld across 1.2B–7B backbones (LFM2.5, Qwen3, Mistral) in a zero-shot transfer setting.

Hi everyone,

I have been reading about Distributional Maximum a Posteriori Policy Optimization (DMPO), especially in the context of the DeepMind bipedal robot soccer paper, and I am trying to understand when one would practically prefer it over SAC.

My current understanding is:

• SAC is a strong off-policy continuous-control baseline.
• It directly optimizes the actor using an entropy-regularized objective.
• It is widely implemented, easier to find baselines for, and generally very strong in simulation.

On the other hand, DMPO seems to use a more structured actor update.

So my interpretation is that DMPO is more like: conservatively update the actor by matching kl divergence from old policy

whereas SAC is more like: mantain entropy and more aggressive updates of actor

I understand why DMPO might be attractive for real-world robotics, since conservative policy updates can reduce dangerous or unstable behavior. But suppose real-world deployment is not the issue, and all trials are in simulation.

In that case, when would you still prefer DMPO over SAC?

For example, would DMPO be more attractive in tasks where:

• the policy is very sensitive to sudden changes?
• the critic is noisy or easy to exploit?
• the task involves contact-rich dynamics?
• the return distribution is multi-modal?
• preserving partially learned behaviors matters?
• coordination between multiple agents is fragile?

Or would you generally just use SAC unless DMPO clearly performs better in ablations?

I am especially interested in practical opinions from people who have tried MPO/DMPO-style algorithms. In what kinds of environments did they outperform SAC, and where did SAC remain the better choice?

Thanks

Hi all, I wanted to get some insights on solving a problem that I'm trying to model as a bandit. I'm fairly new to the subject, so if I'm saying nonsensical things, please explain. Basically, the idea is that pulling an arm gets you a reward, but that reward depends on some factors that change, so pulling the same arm again won't give the same reward. I tried to use epsilon greedy, and things sort of make sense. But, if I want to try UCB or Thompson sampling using Gaussian, it is unclear whether it would be appropriate. Because there is no need to keep pulling an arm if its reward is low when it has been tried only a few times. Depending on the reward design, it indicates that this need not be pulled. Arms, as such, may only be occasionally visited (like in epsilon). So, would this sort of behavior only be like a cold-start problem, and would Thompson eventually learn not to pick it? But how soon would that eventually be? I would appreciate any insights, and I can clarify more if needed, thanks!

Neuroscience question that motivated this: can the kind of learning rules we actually see in the brain; Hebbian plasticity, predictive coding, distributional dopamine signals, be sufficient for a real control task?

I tested this on Pong with a fully backprop-free agent:

• Predictive Coding (Rao & Ballard 1999) for visual feature learning
• Distributional Hebbian plasticity for value estimation, inspired by Dabney et al. 2020 (the finding that dopamine neurons encode a full distribution over future reward, not just a scalar)

Results: BioAgent reaches 57% vs. PPO's 59%. Close, but self-play training exposed a hard problem: Hebbian rules that adapt fast also forget fast under non-stationary opponent dynamics. The plasticity– stability dilemma shows up immediately.

The dopamine-inspired distributional encoding helped stability compared to a scalar baseline, which I found interesting because it suggests the distributional coding might have a functional role beyond just representing uncertainty.

Code: github.com/nilsleut/Biologically-Plausible-RL-Plays-Pong

Curious what people think about the plasticity–stability angle: Is there a biological mechanism for stabilising Hebbian rules under non-stationarity that I'm missing?

What y'all think of the new "agentic" era, pay 200$ to Anthropic to automate a simple task, I really like the idea of automation with reasoning models, but it seems that now everyone can do one, I don't feel comfortable in the current market is like a dystopia,

As a reinforcement learning enthusiast in this sub, do you think this is the lowest moment of humanity? (I do),
How much time do you think this "era" is going to exist? Is it forever?

I am really sad with 2026 honestly, I just think in the line of "The Incredibles":

And when everyone is super...  no one will be!

hey guys I’m new to this whole subject. As the title says I need help upgrading my GPU.

I’m working on my capstone mechanical engineering project, a quadrupedal robot. I decided a few weeks ago that it needed to be trained using Isaac lab. Currently I have isaac sim 6 and isaac lab 3 in a container on my laptop with a 2070.

I’m switching to a desktop but what do you guys think is a better GPU for this software, 3060 12gb or 3080 10gb?

I recently joined Alignerr for a different technical role and noticed they’re looking for people with hands-on MuJoCo / robotics simulation / reinforcement learning experience.

The role seems best suited for people who have worked with MuJoCo, MJCF/XML, Gymnasium/dm_control, reward shaping, PPO/SAC/TD3, physics debugging, and robot control.

It’s remote contractor work. I don’t want to oversell it because project availability can vary, but the listed rate is high and it may be worth checking out if you already have this background.

I have a referral link, but only reach out if you genuinely have MuJoCo/RL experience — this probably isn’t a beginner-friendly role.

Hey folks,

I'm in the midst of some interview prep / learning RL (right now working through spinningup, trying to code/derive some algos from scratch, and building a few example projects) somewhat from scratch. I've found that having accountability is really helpful for making sure progress is made.

Anyone in the same boat who wants an accountability partner? I imagine daily/regular checkins, progress on learning/projects (aka a mini "build in public"), feedback on each others plans, and even some collaboration.

Thanks and If so, DM me!

Running a Rainbow DQN ablation on Snake (C51 + dueling + noisy nets). When I evaluated checkpoints with noise off (mean weights, sigma zeroed out, the standard approach), the scores were all over the place. Some checkpoints averaged 78, others averaged 18. Training curve at those same points was perfectly stable.

First instinct was a bug. Checked everything. It wasn't.

The worst case was at ep450K. Deterministic eval produced a bimodal distribution: ~25% of episodes scored near zero, ~75% scored above 80. The average was 59 but that number is meaningless with two separate peaks and nothing in between.

What's happening: the mean-weight policy has traps. Game states where Q-values for two actions are nearly identical. Without noise, the agent picks the same action every time. If it's the wrong one, it loops and dies. 25% of starting states consistently hit these traps.

Same checkpoint, same seeds, noise turned back on: bimodal failure mode vanished entirely. p25 jumped from 2 to 59. Average went from 59 to 73. Std dropped from 42 to 26. This held at every checkpoint from ep50K through ep450K. Stochastic eval beat deterministic eval across the board.

The noise isn't residual exploration overhead. The agent learned a policy where the sigma values are functional. They provide just enough Q-value perturbation to prevent degenerate action loops. Zero them out and you get a policy that's strictly worse than what the agent actually learned.

Snake makes this especially acute because a single wrong turn at length 100+ is immediately fatal. The deterministic traps are lethal in a way they wouldn't be in more forgiving environments.

One caveat: at one very late checkpoint where sigma had grown extremely large, stochastic eval finally dropped below deterministic. There's a productive zone for noise magnitude, and past it the noise becomes destructive. So it's not "always evaluate with noise." It's "don't assume deterministic eval is automatically the ground truth."

Has anyone else seen this kind of eval divergence with noisy nets? Curious whether it's specific to tight spatial environments like Snake or shows up more broadly.

Hi everyone, I have a question about planning my reinforcement learning studies.

I'm currently a sophomore majoring in a non-CS field. My math background includes calculus, probability and statistics, linear algebra, and some mathematical analysis. I want to start learning reinforcement learning, but according to many recommendations, it seems I may also need additional math courses such as ODEs, real analysis, stochastic processes, etc.

Is that really necessary at my current stage? Or would it be better to learn those topics along the way?

I'd also appreciate any suggestions about how to study reinforcement learning itself (courses, prerequisites, learning path, etc.). So far, the only programming language I’m comfortable with is Python.

RL is powerful enough to train superhuman policies, especially in video games. But is there any research on how to leverage RL's policy/value networks to improve human training speed? How can we apply behavioral cloning to humans?

Past research has shown that simply providing a human with optimal moves doesn't improve their pattern recognition or performance, it only increases their reliance on the feedback, making them worse.

Humans use some form of RL to learn motor skills and are more sample-efficient than algorithms. So, using guidance from expert policies, we can teach humans to learn along optimal trajectories, reducing time wasted in exploration.

Surely, with the help of value predictions, one can determine whether an action was suboptimal, helping solve the credit assignment problem. But what are the optimal ways to signal that to a human(e.g., either provide a number on the screen, display red/green colors, or perhaps electrocute them?)

Is there a learning framework that closes the loop of perception and control, combining SLAM with RL?