u/Public-Journalist820

I experimented with using Reinforcement Learning instead of scripted game AI for a football game.

Each player in this clip is running the same trained policy. During training, the agent never played against opponents, it simply learned how to play football. During inference, I assigned copies of that policy to opposing teams and let them play.

The result was surprisingly entertaining. They competed for possession, attacked, occasionally defended, and even scored the occasional own goal, all without behavior trees or finite state machines.

This was built using RL3, a browser-based Reinforcement Learning playground I've been developing. Curious to hear what game developers think about using RL for gameplay AI instead of traditional approaches.

Video: https://youtu.be/1wRA4KJuMVE?si=oMZ\_Ru5Q9F8J094V 

I trained a single football policy for about 1 million PPO timesteps. During training, the agent never played against an opponent, it simply learned how to play football in an empty environment.

YT Link: https://youtu.be/1wRA4KJuMVE?si=oMZ\_Ru5Q9F8J094V

During inference, I instantiated multiple copies of that same policy and assigned them to opposing teams. I wasn't expecting much, but the result was surprisingly entertaining. The agents naturally competed for possession, took shots, occasionally defended their own goal, and even made mistakes like own goals.

I never explicitly trained defensive behavior or multi-agent coordination. It seems that simply optimizing the objective of scoring goals was enough for some defensive behaviors to emerge when the same policy was placed in a competitive setting.

The video is one of those matches. Curious to hear what others think about this kind of emergent behavior and whether you've observed something similar when deploying a single-agent policy in a multi-agent setting.

After ~15 months of development, my final year project is finally finished.

RL3 is a no-code reinforcement learning playground that runs entirely in the browser for authoring environments and supports server-side deep RL training.

The goal was simple: make reinforcement learning easier to learn without requiring users to write code or set up complicated environments.

The workflow is:

- Design an environment using a drag-and-drop editor.

- Create the reward function visually using a behavior graph.

- Assign graphs to one or more agents.

- Select the training algorithm and start training.

For browser-based training, RL3 supports tabular Q-learning (using state discretization to keep the Q-table manageable).

For deep RL, the authored environment is recreated on the server in PyBullet, where PPO training is available. I'm currently working toward MAPPO support. Multiple-agent inference already works today through shared policies.

Some features include:

- Save and resume training checkpoints.

- Modify environments or reward graphs and continue training (curriculum learning).

- Share environments and trained models with friends.

- Reuse someone else's environment or policy as a starting point.

- Multiple built-in behaviors such as:

- Navigation

- Object collection

- Holding items

- Depositing objects

- Destroying obstacles

- Opening gates with collected keys

- Obstacle avoidance

- Football behaviors

One thing I'm particularly excited about is the football environment. My long-term vision is to let people publish their trained agents and challenge others to either:

- play against them manually, or

- train their own agents to compete.

Eventually I'd like to expand this to other game-like environments (basketball, hide-and-seek, maybe even cricket).

Internally, RL3 uses a behavior/state machine so that only the observations relevant to the current skill are exposed during training. For example, an agent learning to collect objects isn't distracted by observations needed for later tasks like opening gates. This makes learned skills much more reusable when building longer behavior chains.

The project was developed over the past 15 months, mostly by myself, as my final year project. One thing that motivated me was realizing how inaccessible reinforcement learning still feels compared to other areas of AI. Here in Pakistan, RL isn't commonly taught, and most AI discussions revolve around LLMs. I wanted to build something that lowers the barrier to entry and hopefully encourages more people to experiment with RL.

I'll include the deployment link below. The public deployment currently supports Q-learning. PPO requires GPU-backed training pods, which I can't afford to keep running continuously, but if anyone wants to try PPO, feel free to DM me and I'll spin up the training service.

YouTube: https://youtu.be/V5d99hOM5ew?si=JhhZLP3hO-pb69rr

Application Link: https://rl-playground-beta.vercel.app/signing-in

I'd genuinely appreciate feedback from the RL community especially on the overall idea, architecture, and where you think a platform like this could be improved.

I’ve been working on a web-based RL Playground using Three.js on the frontend and Gymnasium + PyBullet + PPO (Stable-Baselines3) on the backend.

So far I have successfully trained:

•	Navigation to a target

•	Coin finding

•	Coin collection

The latest model can navigate toward a coin and perform the collect action when within range.

For my FYP, the expectation is not necessarily many separate agents, but rather an agent capable of executing a longer sequence of interactions (5+). Demo date is 17th June.

Proposed Long-Horizon Task

I’m considering a task chain like:

Find Coin

↓

Collect Coin

↓

Find Deposit

↓

Deposit Coin

↓

Open Gate

↓

Destroy Obstacle

↓

Find Target

↓

Interact With Target

The idea is to train individual abilities through curriculum learning and then combine them into a single policy.

Observation Space Design

Initially I was giving each capability its own Finder observations:

Coin:

[dist, side, depth, in_radius]

Deposit:

[dist, side, depth, in_radius]

Target:

[dist, side, depth, in_radius]

Destroyable:

[dist, side, depth, in_radius]

This started becoming repetitive.

Instead I’m considering introducing a behavior state machine that determines the current objective.

For example:

if holding == 0:

current_goal = COIN

elif deposited == 0:

current_goal = DEPOSIT

elif gate_open == 0:

current_goal = GATE

elif destroyable_destroyed == 0:

current_goal = DESTROYABLE

else:

current_goal = TARGET

The policy would then only receive observations for the active goal.

Proposed Observation Space

# Active Goal Finder

goal_distance

goal_side_signal

goal_depth_signal

goal_in_radius

# Progress State

holding

items_collected

item_deposited

gate_open

destroyable_destroyed

# Goal Indicator

goal_is_coin

goal_is_deposit

goal_is_gate

goal_is_destroyable

goal_is_target

# Navigation

obs_front

obs_left

obs_right

is_blocked

Total is roughly 18-20 dimensions.

The idea is that the policy always sees:

Where is my current objective?

Am I close enough to interact?

What phase of the task am I currently in?

instead of receiving separate direction vectors for every object in the world.

Curriculum Plan

Current thought process:

Stage 1

Find Coin

Stage 2

Collect Coin

Stage 3

Find Deposit

Stage 4

Deposit Coin

Stage 5

Open Gate

Stage 6

Destroy Obstacle

Stage 7

Find Target

Stage 8

Combine everything into a single policy

Each stage would start with fixed spawns and gradually move toward randomized spawns.

Main Question

For those who have trained PPO agents on long-horizon tasks:

1.	Does the active-goal observation design seem reasonable?

2.	Would you expose only the current objective or all object directions simultaneously?

3.	Any obvious pitfalls before I commit to this curriculum approach?

After 9 months of work I finally got my first successful run in a simple RL environment where the agent learns to find a target 🎉

I’m still validating more SARL scenarios, but I’m now thinking ahead toward MARL and wanted some advice on architecture and trainer choice.

Current RL engine structure:

1.	SimulationEngine

•	Handles both logic and physics orchestration

•	Calls the other layers internally

2.	EnvironmentEngine

•	Handles environment logic

3.	BulletWorld

•	Builds and manages the PyBullet world

I also have a Gymnasium wrapper:

env = GymWrapper(simulation_engine)

which exposes clean reset() and step() APIs for SB3.

The thing is: internally SimulationEngine already works with dictionary-based outputs:

{

"agent_1": observation,

"agent_2": observation

}

For SARL + Gymnasium I transform this into something meaningful for SB3.

But from what I understand, PettingZoo naturally expects agent-keyed dictionaries, which makes me think my current architecture could fit MARL pretty neatly without major redesign.

My main concern is the trainer side.

SB3 + Gymnasium has been incredibly straightforward and I already have experience with it.

But for:

PettingZoo + ???

I’m stuck.

Initially I was considering RLlib because it seems to be the common answer, but I honestly don’t have the time/energy for a steep learning curve if there are cleaner alternatives.

I’m mainly interested in MAPPO and similar MARL algorithms.

Questions:

•	What trainer stack are people using with PettingZoo nowadays?

•	RLlib vs BenchMARL vs AgileRL vs something else?

•	If you were building this from scratch today, what would you choose?

Any suggestions or experiences would be really appreciated.

Hey guys,

I’m building a reinforcement learning playground as part of my final year project (FYP), mainly aimed at helping students/teachers learn RL visually, and I’d love to get feedback.

Core ideas:

🔹 Capability System (MOVEABLE, FINDER, NAVIGATOR, etc.)

Agents are composed from capabilities instead of hardcoded environments.

Each capability defines:

•	Action space

•	Observations (OBS space)

•	State contributions

This makes environments modular and easier to reason about.

🔹 Visual Reward Design (Graph-based)

Reward functions are built as graphs:

•	Conditional nodes (distance checks, radius, etc.)

•	Logical flow

•	Rewards / penalties / termination

No code, everything is visual.

🔹 Assignment Panel (Agent ↔ Graph ↔ Algo)

•	Bind one or more agents to a behavior graph

•	Configure training (PPO supported)

•	Shared policy works naturally at inference, spawning agents with the same capabilities reuses the learned policy

🔹 Tech Stack / Architecture

•	Frontend: Three.js + Rapier.js

•	Training: PyBullet + Gym + Stable-Baselines3 (PPO)

•	Inference: Remote PPO controller via WebSocket

•	Also includes a client-side tabular Q-learning option (more for learning/demo, limited scalability)

🔹 LLM-Assisted Workflow

•	Suggests reward function improvements while designing

•	Explains trained model behavior + parameters during analysis

🔹 What’s next

•	Proper multi-agent support (currently structuring toward it)

Where I need help / feedback:

One thing I’m still figuring out properly is:

👉 How to define good observation spaces (OBS) for different capabilities in a way that’s both generalizable and intuitive.

Would love input on that specifically.

If this looks interesting, I’d be happy to share access for testing. Also open to any feedback / criticism especially around abstractions and usability.

Thanks 🙏