A central challenge in multi-agent reinforcement learning is the induction of coordination between agents of a team. In this work, we investigate how to promote inter-agent coordination using policy regularization and discuss two possible avenues respectively based on inter-agent modelling and synchronized sub-policy selection. We test each approach in four challenging continuous control tasks with sparse rewards and compare them against three baselines including MADDPG, a state-of-the-art multi-agent reinforcement learning algorithm. To ensure a fair comparison, we rely on a thorough hyper-parameter selection and training methodology that allows a fixed hyper-parameter search budget for each algorithm and environment. We consequently assess both the hyperparameter sensitivity, sample-efficiency and asymptotic performance of each learning method. Our experiments show that the proposed methods lead to significant improvements on cooperative problems. We further analyse the effects of the proposed regularizations on the behaviors learned by the agents.
This paper proposed two approaches to encourage cooperation among multi-agents under the *centralized training decentralized execution* framework. The main contribution of this paper is that they propose to allow agents to predict the behavior of others and introduce this prediction loss into the RL learning objective. One approach is teammate-regularization where each agent predicts the behavior of all others (and therefore makes the total model complexity increasing quadratically with the number of agents), while the other is a centralized coach-regularization. The performance of these two approaches are compared with MADDPG and its two variants on 4 games. Experiments show that their approaches surpass a few baselines in certain game settings. However, the novelty of this algorithm is not strong, given that the similar idea of 'predicting the behavior of other agents' can be found in related work with discrete action spaces. Experimental results also show unstable advantages of their approaches.
Methodology:
Team-reg: One main difference between this approach and Jaques work: * Intrinsic social motivation via causal influence in multi-agent*  (or other MARL algorithms for discrete action spaces) is that in this work, each agent predict the action of other peers instead of the policy, since this algorithm is only based on DDPG with continuous action space. However, It seems that this idea is transferrable to discrete action spaces as well, so long as we change the MSE loss between actions to KL loss between policy over states. The novelty of this approach is not strong. Also, one hypothesis on which their work is based is that promoting agents’ predictability could foster such team structure and lead to more coordinated behaviors. Does predictability really improve cooperation? Could it be the other way around? Experiment results in the right-below penal (CHASE Game ) and left-above penal (SPREAD game) of Figure 5 actually strength my concern. 
Coach-reg: Descriptions of the coach regularization approach is quite vague. I doubt the motivation and effectiveness of this approach. More questions are as follows:
* What is the role of the policy mask in deriving a better action? Is it just a dropout before the activation layer with a fixed $p$ proportion (which is related to the choice of $K$)? If so, this is like you first assume the policy network is somewhat overfitting, then alleviate this issue by letting the coach adjust which part to keep and which part to drop.  How would different $K$ values affect the performance?
* Does this framework really improve cooperation? Intuitively, all agents will finally reach a point where they agree on the same policy mask distribution (which is the one generated by the coach). How does the same policy mask lead to an improvement in cooperation? Empirical or theoretical explanations are strongly needed in the main results, not in the appendix section.
Experiments:
- All variants of MADDPG in the experiments are weak baselines, assuming that $MADDPG + sharing$ means all agents share the same policy and Q-function. However, since this algorithm only works for continuous action space, available relate work to compare with is quite limited. All environments only include two agents. Experiments with more than two agents should be implemented. 
- Even for only 3 games (excluding the adversarial game setting), both branches failed to beat the *sharing* baseline for the CHASE game. For the ablation study, *agent-moduling* even works better than *TeamReg* for the CHASE game, so as the case when *policy mask* beats *CoachReg* in the SPREAD game (Figure 5). They seem to show that under these two proposed frameworks, the predictability of agents does not always encourage cooperation, at least for 2 out of 4 game settings mentioned in this section.
- This work makes a lot of effort on the hyper-parameter tuning part, which however does not provide a systematic solution to the tradeoff between improving predictability and maximizing reward.