DQN vs PPO/SAC for obstacle avoidance in CARLA
Hi everyone,
I’m working on a bachelor research project on using reinforcement learning for dynamic obstacle avoidance in the CARLA simulator.
My current setup is:
- CARLA simulator
- LiDAR observations
- Raw 3D point cloud reduced to a 1D array of 360 minimum-distance values
- Stable-Baselines3
- DQN
- Discrete action space with 36 steering/throttle combinations
- Custom Gym environment
- The goal is to avoid dynamic obstacles while continuing to drive forward
I chose DQN partly because my action space is discrete, but I’m interested in whether this was actually a sensible choice for this type of problem.
I’d especially appreciate feedback on these points:
1. DQN vs PPO/SAC
Given the 360-value LiDAR observation and discrete steering/throttle action space, would you consider DQN a reasonable baseline?
Would you expect PPO or SAC to have significant advantages here, and if so, why? I’m particularly interested in whether the continuous-action capabilities of algorithms such as SAC would actually be useful enough to justify changing the action space.
2. Reward design
How important is reward engineering in obstacle-avoidance tasks like this?
My main issue is finding a balance between rewarding forward progress and penalizing collisions/unsafe behaviour without encouraging unwanted behaviour such as simply stopping to avoid obstacles.
Are there particular reward-design principles or common failure modes I should be aware of?
3. CARLA and RL
For research into RL-based autonomous driving, do you consider CARLA a suitable environment, or are there other simulators/frameworks you would recommend?
I liked using CARLA but it was pretty heavy which made it so I didn't run as many training runs
Thanks in advance for any feedback or suggestions!