Learning to Communicate with Deep Multi-Agent Reinforcement Learning
We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate endto-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.
The tasks that we consider are fully cooperative, partially observable, sequential multi-agent decision making problems. All the agents share the goal of maximising the same discounted sum of rewards. While no agent can observe the underlying Markov state, each agent receives a private observation correlated with that state. In addition to taking actions that affect the environment, each agent can also communicate with its fellow agents via a discrete limited-bandwidth channel. Due to the partial observability and limited channel capacity, the agents must discover a communication protocol that enables them to coordinate their behaviour and solve the task.
We focus on settings with centralised learning but decentralised execution. In other words, communication between agents is not restricted during learning, which is performed by a centralised algorithm; however, during execution of the learned policies, the agents can communicate only via the limited-bandwidth channel. While not all real-world problems can be solved in this way, a great many can, e.g., when training a group of robots on a simulator. Centralised planning and decentralised execution is also a standard paradigm for multi-agent planning [1,2].
To address this setting, we formulate two approaches. The first, reinforced inter-agent learning (RIAL), uses deep Q-learning [3] with a recurrent network to address partial observability. In one variant of this approach, which we refer to as independent Q-learning, the agents each learn their own network parameters, treating the other agents as part of the environment. Another variant trains a single network whose parameters are shared among all agents. Execution remains decentralised, at which point they receive different observations leading to different behaviour.
The second approach, differentiable inter-agent learning (DIAL), is based on the insight that centralised learning affords more opportunities to improve learning than just parameter sharing. In particular, while RIAL is end-to-end trainable within an agent, it is not end-to-end trainable across agents, i.e., no gradients are passed between agents. The second approach allows real-valued messages to pass between agents during centralised learning, thereby treating communication actions as bottleneck connections between agents. As a result, gradients can be pushed through the communication channel, yielding a system that is end-to-end trainable even across agents. During decentralised execution, real-valued messages are discretised and mapped to the discrete set of communication actions allowed by the task. Because DIAL passes gradients from agent to agent, it is an inherently deep learning approach.
Experiments on two benchmark tasks, based on the MNIST dataset and a well known riddle, show, not only can these methods solve these tasks, they often discover elegant communication protocols along the way. To our knowledge, this is the first time that either differentiable communication or reinforcement learning with deep neural networks has succeeded in learning communication protocols in complex environments involving sequences and raw images. The results also show that deep learning, by better exploiting the opportunities of centralised learning, is a uniquely powerful tool for learning such protocols. Finally, this study advances several engineering innovations that are essential for learning communication protocols in our proposed benchmarks.
Given the topic of our paper, we focus on related work that deals with the cooperative learning of communication protocols. Out of the plethora of work on multi-agent RL with communication, e.g., [4][5][6][7], only a few fall into this category. Most assume a pre-defined communication protocol, rather than trying to learn protocols. One exception is the work of Kasai et al. [7], in which tabular Q-learning agents have to learn the content of a message to solve a predator-prey task with communication.
Another example of open-ended communication learning in a multi-agent task is given in [8]. Here evolutionary methods are used for learning the protocols which are evaluated on a similar predator-prey task. Their approach uses a fitness function that is carefully designed to accelerate learning. In general, heuristics and handcrafted rules have prevailed widely in this line of research. Moreover, typical tasks have been necessarily small so that global optimisation methods, such as evolutionary algorithms, can be applied. The use of deep representations and gradient-based optimisation as advocated in this paper is an important departure, essential for scalability and further progress. A similar rationale is provided in [9], another example of making an RL problem end-to-end differentiable.
Unlike the recent work in [10], we consider discrete communication channels. One of the key components of our methods is the signal binarisation during the decentralised execution. This is related to recent research on fitting neural networks in low-powered devices with memory and computational limitations using binary weights, e.g. [11], and previous works on discovering binary codes for documents [12].
The paper considers problems involving multiple agents in an environment with a shared goal, and attempts to learn strategies to solve such collaborative tasks. The novelty of the work is to introduce differentiable real-valued message-passing between agents, so that their communication can be trained end-to-end in order to solve the collaborative tasks.  The resulting Differential Inter-Agent Learning (DIAL) technique has been thoroughly evaluated on complex games like the switch riddle and MNIST digit recognition games to evaluate its utility and convergence to optimal policy. A good empirical study has been carried out to compare DIAL and RIAL in the presence and absence of parameter sharing and the study shows encouraging results for DIAL.
Though the paper contains a very thorough experimental evaluation of the suggested DIAL technique for multi-agent settings and the reviewer understands that it would have taken a lot of time and effort to set up and evaluate the experiments, the paper does not make a very novel contribution.  It is clear that idea of shared memory and passing message gradients between agents would speed up learning and help to find the optimal policy faster, but this might not be a very natural way to do it. For instance, humans working in teams do not have shared memories. Also, for humans, messages from other humans are a part of their observation at each time step, rather than separate signals which are treated specially as messages and optimized differently than the rest of the observation.  The idea of passing message gradients is certainly useful to have trainable message protocols while training a set of agents to perform a repetitive task, but doesn't offer much insight or useful interpretation as to how humans perform tasks in teams.  Furthermore, experimental evaluation has been done for very small instances of the games involved, e.g. only for n = 3 and 4 for the switch riddle. It is probably due to the size of the policy space involved. But this is exactly what is needed from a good reinforcement learning algorithm - combatting huge state spaces with intelligent exploration instead of random exploration. Even while going from n=3 to n=4, there is significant performance degradation involved in the sense that DIAL (with parameter sharing) fails to find the optimal policy despite the increase in number of epochs from ~5000 to ~50000. This somewhat reflects the non-scalability of the algorithm for many agents. Moreover since the paper relies highly on experimental evaluation, it might be useful to try more challenging and bigger tasks to evaluate the performance and scalability of the DIAL technique.  Apart from this, a language and grammar check can improve the presentation somewhat. Also avoid repetitions, for example in the last paragraph of section 5.2. Lastly, at some places relaxing the notations a bit can enhance readability. For instance, in the color-digit MNIST in section 6.3, the reward r(a) was quite hard to interpret. It is also not clear what a_2^a is, since it hasn't been described in that section (perhaps the authors meant u_2^a).  EDIT AFTER REBUTTAL: The authors have clarified some of my raised concerns, regarding scalability of the algorithm and its contrast to how groups of humans accomplish such tasks. I am convinced with the response and consequently I am improving my ratings slightly to reflect the same.