title: Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
abstract: A fascinating aspect of nature lies in its ability to produce a large and diverse collection of organisms that are all high-performing in their niche. By contrast, most AI algorithms focus on finding a single efficient solution to a given problem. Aiming for diversity in addition to performance is a convenient way to deal with the exploration-exploitation trade-off that plays a central role in learning. It also allows for increased robustness when the returned collection contains several working solutions to the considered problem, making it well-suited for real applications such as robotics. Quality-Diversity (QD) methods are evolutionary algorithms designed for this purpose. This paper proposes a novel algorithm, QD-PG, which combines the strength of Policy Gradient algorithms and Quality Diversity approaches to produce a collection of diverse and high-performing neural policies in continuous control environments. The main contribution of this work is the introduction of a Diversity Policy Gradient (DPG) that exploits information at the time-step level to thrive policies towards more diversity in a sample efficient manner. Specifically, QD-PG selects neural controllers from a MAP-Elites grid and uses two gradient-based mutation operators to improve both quality and diversity, resulting in stable population updates.  Our results demonstrate that QD-PG produces collections of diverse solutions that solve challenging exploration and control problems while being two orders of magnitude more sample efficient than its evolutionary competitors.
authors: Thomas PIERROT Valentin Macé Geoffrey Cideron Karim Beguir Antoine Cully Olivier Sigaud Nicolas Perrin
hIndex: Antoine Cully 17