We re-think the Two-Player Reinforcement Learning (RL) as an instance of a distribution sampling problem in infinite dimensions. Using the powerful Stochastic Gradient Langevin Dynamics, we propose a new two-player RL algorithm, which is a sampling variant of the two-player policy gradient method. Our new algorithm consistently outperforms existing baselines, in terms of generalization across differing training and testing conditions, on several MuJoCo environments.
Summary:
The paper considers the task of learning robust policies. Specifically, they focus on the noisy-robust (NR) variant of the action robust framework proposed in [1]. As noted in [1], as strong duality does not hold in the NR variant it can not be solved using deterministic policies (pure strategies).
The authors propose to combine SGLD with DDPG, to maintain a distribution over deterministic policies (essentially a mixed strategy) - hence overcoming these issues.
The results look promising, outperforming [1] in all tested domains.
Review:
Overall this seems like the first approach of using Bayesian learning in order to reach mixed Nash equilibrium in RL. This is super-important, since many problems can be formulated as zero-sum games for which the solution is not necessarily a pure strategy.
As [2] have already shown the ability of this concept to find mixed equilibria in GANs, the main novelty is the introduction to RL.
As this work does not introduce new theory or a dramatic new concept, I feel that the acceptance of this work lies mainly on the empirical side. In my opinion, the experiments need to be more convincing - for instance, include additional domains (such as the Inverted Pendulum which was a failure case in [1]) and additional forms of robustness (e.g., the probabilistic-robust variant from [1] which is based on deterministic strategies and was shown to work better, and RARL [3] which test robustness to external disturbances).
An addition option is to build a low-dimensional toy problem in which the exact solution is known. This will enable you to show that while the naive solution in [1] either does not converge or converges to sub-optimal solutions, the SGLD approach is capable of finding superior solutions (hopefully the global optimum).
The idea is in the right direction. However, in my opinion, it is not there yet and is thus not ready for ICLR.
--- Post Rebuttal ---
I stand by my original assessment. I feel that such work needs to be more convincing. I for one would feel more confident had the authors provided simpler experiments in which they show that their approach indeed converges to the mixed Nash equilibria while the NR-DDPG approach from [1] does not.
I did overall like this direction and I believe Robustness in RL is very important.
[1] Action Robust Reinforcement Learning and Applications in Continuous Control - Tessler et al. 2019
[2] Finding mixed nash equilibria of generative adversarial networks - Hsieh et al. 2019
[3] Robust Adversarial Reinforcement Learning - Pinto et al. 2017