title: A Cramér Distance perspective on Non-crossing Quantile Regression in Distributional Reinforcement Learning
abstract: Distributional reinforcement learning (DRL) extends the value-based approach by estimating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performances. Quantile-based methods like QR-DQN project arbitrary distributions into a parametric subset of staircase distributions by minimizing the 1-Wasserstein distance, however,  due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Recently, monotonicity constraints on the quantiles have been shown  to improve the performance of QR-DQN for uncertainty-based exploration strategies. The contribution of this work is in the setting of fixed quantile levels and is twofold.  First, we prove that the Cramér distance yields a projection that coincides with the 1-Wasserstein one and that, under monotonicity constraints, the squared Cramér and the quantile regression losses yield collinear gradients, shedding light on the connection between these important elements of DRL. Second, we propose a novel non-crossing neural architecture that allows a good training performance using the Cramér distance, yielding significant improvements over QR-DQN in a number of games of the standard Atari 2600 benchmark.
authors: Alix Lhéritier Nicolas Bondoux
hIndex: Nicolas Bondoux 3