Intrinsically motivated reinforcement learning aims to address the exploration challenge for sparse-reward tasks. However, the study of exploration methods in transition-dependent multi-agent settings is largely absent from the literature. We aim to take a step towards solving this problem. We present two exploration methods: exploration via information-theoretic influence (EITI) and exploration via decision-theoretic influence (EDTI), by exploiting the role of interaction in coordinated behaviors of agents. EITI uses mutual information to capture the interdependence between the transition dynamics of agents. EDTI uses a novel intrinsic reward, called Value of Interaction (VoI), to characterize and quantify the influence of one agent's behavior on expected returns of other agents. By optimizing EITI or EDTI objective as a regularizer, agents are encouraged to coordinate their exploration and learn policies to optimize the team performance. We show how to optimize these regularizers so that they can be easily integrated with policy gradient reinforcement learning. The resulting update rule draws a connection between coordinated exploration and intrinsic reward distribution. Finally, we empirically demonstrate the significant strength of our methods in a variety of multi-agent scenarios.
This paper studies the problem of designing effective exploration strategies in multi-agent domains. The key idea is to define one agent's exploration in terms of its interactions with other agents. This leads to two auxiliary exploration objectives, which measure how one agent's actions affect the dynamics and value of another agent's actions. The paper does an admirable job comparing the proposed method against a number of baselines, where the proposed method performs significantly better. Visualizations and ablation experiments nicely illustrate the contributions of various components of the method.
I am leaning towards accepting the paper. To the best of my knowledge, the broad idea of applying information theory to multi-agent exploration, in addition to the specific instantiation described in the paper, is novel. I expect that this paper will encourage future work to explore more problems in this area. The experiments are quite thorough. My main reservation is a lack of comparisons to single agent exploration methods. As noted in Section 3, we can view multi-agent domains as just a special type of single agent domain. How would curiosity-based exploration, such as [Burda 2018, Pathak 17], or mutual information-based exploration, such as [Gregor 16, Eysenbach 18, Achaim 18], compare to the proposed method?
I have a few reservations about the clarity of presentation, but I think those are easily addressed. My remaining concern is that the results are on somewhat toy tasks, but I think that is par for this area of research.
Overall, I would strongly argue for accepting this paper if comparisons to single-agent exploration methods were added. I would consider decreasing my review if another reviewer found quite similar prior work, or if significant bugs were found in the mathematical derivation (I have not carefully checked all the proofs in the appendix.).
Minor comments
* "transition-dependent" -- what does this mean?
* "while tend" -- missing a subject
* "struggle in many real-world scenarios with sparse rewards" -- please add a citation
* "intrinsic value function of agent i, I_{-i|i}^\pi is \beta > 0 is a weighting" -- I think part of this sentence was accidentally deleted.
* Eq 5: What is the difference between I and MI?
* "We call …" -- What is the a_2^V0I^\pi_{-i|i} term?
* Nitpick: Use `` for the start of quotes
* Appendix B1: How is Eq 22 obtained from Eq 21?
-------------------UPDATE AFTER AUTHOR RESPONSE---------------------
The authors have done a great job address my two concerns (similarity to prior work and empirical comparisons with single-agent exploration). I therefore increase my vote to "accept."