LEARNING TO SCHEDULE COMMUNICATION IN MULTI-AGENT REINFORCEMENT LEARNING
Many real-world reinforcement learning tasks require multiple agents to make sequential decisions under the agents' interaction, where well-coordinated actions among the agents are crucial to achieve the target goal better at these tasks. One way to accelerate the coordination effect is to enable multiple agents to communicate with each other in a distributed manner and behave as a group. In this paper, we study a practical scenario when (i) the communication bandwidth is limited and (ii) the agents share the communication medium so that only a restricted number of agents are able to simultaneously use the medium, as in the state-of-the-art wireless networking standards. This calls for a certain form of communication scheduling. In that regard, we propose a multi-agent deep reinforcement learning framework, called SchedNet, in which agents learn how to schedule themselves, how to encode the messages, and how to select actions based on received messages. SchedNet is capable of deciding which agents should be entitled to broadcasting their (encoded) messages, by learning the importance of each agent's partially observed information. We evaluate SchedNet against multiple baselines under two different applications, namely, cooperative communication and navigation, and predator-prey. Our experiments show a non-negligible performance gap between SchedNet and other mechanisms such as the ones without communication and with vanilla scheduling methods, e.g., round robin, ranging from 32% to 43%.
# overview
This paper focuses on multi-agent reinforcement learning tasks that require communication between the agents, and further presupposes that the communication protocol is bandwidth constrained and contentious so that a scheduling mechanism is necessary.  To address this they introduce a new learned weighting scheme based scheduler and distributed actor, centralized critic based architecture which is evaluated on a couple of communication driven multi-agent tasks.
The two evaluation tasks had their bandwidth artificially constrained, and SchedNet time to convergence was shown to fall somewhere between having no communication and full communication, and somewhat better than a purely round-robin based scheduling scheme, which doesn't seem particularly informative.  From this it is difficult to assess the significance of the contributions.
# pros
* communication in multi-agent scenarios is an important aspect to consider, and this work shines a spotlight on scenarios in which bandwidth is constrained.
* general presentation fairly clear and easy to read
# cons
* Would have been more impactful to focus experiments on real-world scenarios in which bandwidth is constrained and naturally contentious
# other comments
* pg. 2 related work, suspect you meant to call out Foerster et al 2017b in second reference not Foerster et al 2017a twice.