CaRL: Learning Scalable Planning Policies with Simple Rewards
We investigate reinforcement learning (RL) for privileged planning in
autonomous driving. State-of-the-art approaches for this task are rule-based,
but these methods do not scale to the long tail. RL, on the other hand, is
scalable and does not suffer from compounding errors like imitation learning.
Contemporary RL approaches for driving use complex shaped rewards that sum
multiple individual rewards, \eg~progress, position, or orientation rewards. We
show that PPO fails to optimize a popular version of these rewards when the
mini-batch size is increased, which limits the scalability of these approaches.
Instead, we propose a new reward design based primarily on optimizing a single
intuitive reward term: route completion. Infractions are penalized by
terminating the episode or multiplicatively reducing route completion. We find
that PPO scales well with higher mini-batch sizes when trained with our simple
reward, even improving performance. Training with large mini-batch sizes
enables efficient scaling via distributed data parallelism. We scale PPO to
300M samples in CARLA and 500M samples in nuPlan with a single 8-GPU node. The
resulting model achieves 64 DS on the CARLA longest6 v2 benchmark,
outperforming other RL methods with more complex rewards by a large margin.
Requiring only minimal adaptations from its use in CARLA, the same method is
the best learning-based approach on nuPlan. It scores 91.3 in non-reactive and
90.6 in reactive traffic on the Val14 benchmark while being an order of
magnitude faster than prior work.
68126aa5ec67f8f30bf43f20
Bernhard Jaeger
68126aa5ec67f8f30bf43f21
Daniel Dauner
68126aa5ec67f8f30bf43f22
Jens Beißwenger
claimed_verified
2025-07-09T15:10:50.993Z
66e47eb35c06b7719c0ffde6
/avatars/22166e82fed2019fdfdd46736d259495.svg
Jens Beißwenger
user
Jens21
68126aa5ec67f8f30bf43f23
Simon Gerstenecker
68126aa5ec67f8f30bf43f24
Kashyap Chitta
claimed_verified
2025-07-05T07:54:21.608Z
6489d7aada6bb0138f52027b
https://cdn-avatars.huggingface.co/v1/production/uploads/6489d7aada6bb0138f52027b/py4YEgVs8Y-QWzCmpa6Xx.jpeg
Kashyap Chitta
user
kashyap7x
68126aa5ec67f8f30bf43f25
Andreas Geiger
CaRL: Learning Scalable Planning Policies with Simple Rewards
We investigate reinforcement learning (RL) for privileged planning in
autonomous driving. State-of-the-art approaches for this task are rule-based,
but these methods do not scale to the long tail. RL, on the other hand, is
scalable and does not suffer from compounding errors like imitation learning.
Contemporary RL approaches for driving use complex shaped rewards that sum
multiple individual rewards, \eg~progress, position, or orientation rewards. We
show that PPO fails to optimize a popular version of these rewards when the
mini-batch size is increased, which limits the scalability of these approaches.
Instead, we propose a new reward design based primarily on optimizing a single
intuitive reward term: route completion. Infractions are penalized by
terminating the episode or multiplicatively reducing route completion. We find
that PPO scales well with higher mini-batch sizes when trained with our simple
reward, even improving performance. Training with large mini-batch sizes
enables efficient scaling via distributed data parallelism. We scale PPO to
300M samples in CARLA and 500M samples in nuPlan with a single 8-GPU node. The
resulting model achieves 64 DS on the CARLA longest6 v2 benchmark,
outperforming other RL methods with more complex rewards by a large margin.
Requiring only minimal adaptations from its use in CARLA, the same method is
the best learning-based approach on nuPlan. It scores 91.3 in non-reactive and
90.6 in reactive traffic on the Val14 benchmark while being an order of
magnitude faster than prior work.
Reinforcement learning with a simplified reward design achieves superior performance in autonomous driving tasks by scaling well with larger mini-batch sizes and distributed training.
reinforcement learning privileged planning RL PPO mini-batch size distributed data parallelism CARLA nuPlan Val14 benchmark