Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multiagent systems. However, existing methods have been focusing on learning fully decentralized value function, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows to cut off more than 80% communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndvf.
The paper tackles the collaborative multi-agent RL problem as the problem of finding almost-decentralized value functions, where the loss tries to minimize communications between the agents. The core idea is around maximizing the mutual information between each massage and the existing knowledge of its receiver. This way, redundant messages are naturally removed. The authors then assert an entropy regularization to (almost) prevent the agents from *cheating*. The paper is in general well-written and motivated. There are however certain issues that should be addressed. [The second one is my main issue.]  
1. The term *message* has been used repeatedly but not defined. You should define precisely what you mean by a message in the background section. In particular, it is not clear from the text what a message looks like mathematically. You may also consider giving intuitive examples of how different ways of designing a message alters the behaviour in a given context before start talking about how to optimise them. 
2. Section 3.1 needs a revisit. I assume by "optimal action policy of agent j" you mean "optimal policy of agent j". Formally, an optimal policy is greedy to the optimal value function and is deterministic unless there exist multiple actions with same optimal value at a given state. Therefore, the mutual information is not mathematically well-defined since most of the time the optimal policy is deterministic and does not induce a probability distribution. 
2.1 What is the optimal value function of an agent? The agents are not optimizing their own value function, so even if your complex model converges, it does not imply optimality of agent-level value functions (and they shouldn’t be locally optimal). If by that you mean the agent-level value functions *after* convergence of $Q_tot$, then you need to clarify it to avoid the confusion. Even so, it is still not clear how you conceive the agent-level policies from these value functions. 
2.2. Minor: As for the notation, I found $A_j$ quite strange for a policy; you may want to consider something like $\pi_j$. 
3. If $f_m$ is learnt, then message encoding is not stationary at least in the training time. It may make the training potentially become unstable. Specifically, there is no condition to assure stability of your model. Suggestion: your entropy regularization might be sufficient to induce stability. More discussion on this (or a simple experiment to show if an instability exists and will be alleviated with regularization) would be quite helpful.  
4. If I understand it correctly [and it wasn't clear from the text], the encoder $f_m$ is conditioned on local history, which induces that all the agents must share same history shape (tensor-wise), which in turn means that all the local agents has to have same local state shape (computationally, they should for example have same output/internal-state shape in their neural networks). This sounds like a limitation, and it may only be OK in domains where agents are homogenous.