A Deep Bayesian Policy Reuse Approach Against Non-Stationary Agents
In multiagent domains, coping with non-stationary agents that change behaviors from time to time is a challenging problem, where an agent is usually required to be able to quickly detect the other agent's policy during online interaction, and then adapt its own policy accordingly. This paper studies efficient policy detecting and reusing techniques when playing against non-stationary agents in Markov games. We propose a new deep BPR+ algorithm by extending the recent BPR+ algorithm with a neural network as the value-function approximator. To detect policy accurately, we propose the rectified belief model taking advantage of the opponent model to infer the other agent's policy from reward signals and its behaviors. Instead of directly storing individual policies as BPR+, we introduce distilled policy network that serves as the policy library in BPR+, using policy distillation to achieve efficient online policy learning and reuse. Deep BPR+ inherits all the advantages of BPR+ and empirically shows better performance in terms of detection accuracy, cumulative rewards and speed of convergence compared to existing algorithms in complex Markov games with raw visual inputs.
The paper describes a new version of BPR+, a Bayesian Policy Reuse algorithm for adapting to opponent/teammate behaviors in multiagent settings. It adds two elements: an opponent model, and a distilled policy network, that help improve some shortcomings of the previous versions.  The paper is well-written, the motivation and problems with the previous algorithms are well-articulated. On a positive note, the experiments do a good job of showing the strengths of each of the new elements of Deep BPR+, in three domains, mixing both cooperative and competitve, testing each component separately. It is clear how much each element contributes in these problems. The experiments against online unknown policies is especially valuable and complementary to the motivation.   As a down side, the novelty is somewhat low because the main algorithms already exist, and the new elements here are straight-forward adaptations of ideas from deep learning architectures that have been used in several other places already. After reading the start of Section 3, I was expecting that adding deep learning would mean the ability to scale to larger or more complex domains. That may be true, but it's not made clear by this paper (nor is it motivated as such).  Minor comments: - "LOLA[4] studies the emergence of Nash equilibrium in iterated prisoners' dilemma". This is not quite true; it found *tit-for-tat* in IPD, which is much more surprising. It found the only mixed Nash equilibria in iterated matching pennies, but in any case this was not necessarily intended and there is no proof that LOLA is trying to find an equilibirum.  Questions: - The construction of the distilled policy network sounds very similar to the architecture behind DRON. Can you clarify the main difference(s)? DRON seems to have a mixture-of-experts component used to decide how to switch.. is this also true for DPN?  - Is the Omniscient policy in the experiments exact? (i.e. computed in the tabular form? If so, how was it computed (value iteration?)  Suggested future work: 1. I would encourage the authors to look into more challenging domains either with partial observability or more than two players where the gains from opponent modeling are more clear. 2. On a similar note, I encourage the authors to relate or combine this approach with recent game-theoretic approaches: the overall process resembles fictitious play and/or the oracles algorithm in [1,2], where the DPN can be interpreted as the average or meta-policy (here it acts as a natural baseline to start from, but in self-play, could also provably converge to an equilibrium?). However, there was no explicit opponent modeling in those works, so Deep BPR+ can enhance these methods while potentially keeping the guarantees.   [1] Heinrich & Silver 2016. Deep Reinforcement Learning from Self-Play in Imperfect-Information Games. https://arxiv.org/abs/1603.01121  [2] Lanctot et al. 2017. A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning.  **** Post-rebuttal:   Thank you for the clarifications. I raised my score after the discussion, upon reflecting on the novelty (rectified belief update) and the wording attached to the numbers: I think this could work could be of interest to the growing multiagent learning community at NIPS.