EMERGENT COORDINATION THROUGH COMPETITION
We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, populationbased training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.
This paper introduces a new multiagent research environment---a simplified version of 2x2 RoboSoccer using the MuJoCo physics engine with spherical players that can rotate laterally, move forwards / backwards, and jump.
The paper deploys a fine-tuned version of population-based sampling on top of a stochastic value gradient reinforcement learning algorithm to train the agents.  Some of the fine-tunings used include deploying different discount factors on multiple different reward channels for reward shaping.
The claimed novel contributions of the paper are (1) a new multiagent testbed, (2) a decentralized training procedure, (3) fine-tuning reward shaping, and (4) highlighting the challenges in evaluation in novel multiagent competitive environments.
Overall, my judgment is that the paper is fine, but the authors have not helped me to understand the significance of their contributions.
Taking each in turn:
(1) What is the significance of the new environment?  What unique characteristics make it difficult?  What makes this environment an importantly different testbed or development environment?  The connection to RoboSoccer is motivating but tenuous. The new environment should have particular characteristics that expose problems with past algorithms or offer new challenges existing algorithms have not addressed at all.
(2) Why is it important to have a decentralized training procedure when the authors have control over all the agents?  If it will allow faster training, has the authors' algorithm been demonstrated to accomplish that goal?  
(3) It's hard to evaluate new algorithms when the domain studied is also new. We have no sense for state-of-the-art performance on this domain across a range of algorithms.  The authors conduct a careful ablation study on their new algorithm but do not compare their approach to other classes of algorithms.
(4) The authors indicate that evaluating the quality of an algorithm for a competitive context is hard in absence of established benchmarks---whereas in single-agent or cooperative environments progress can be measured against the goal of the environment, progress in competitive environments requires comparison to approaches that are thought to be good.  Here the authors are themselves pointing out a fundamental problem with introducing new competitive multiagent testbeds, and the authors don't resolve this tension.  Since the main contribution of the work is the environment, it's hard to see how this point the authors themselves make doesn't undermine that central contribution.
Besides other comments mentioned above, a couple other ways to improve the paper would be:
- Clarify why this environment is important to be introducing---what are the unique things that can be studied with this new environment?
- Hold an open competition to get benchmarks created by other teams of researchers
Some minor comments:
- $n_r$ is not defined explicitly in the text as far as I have found
- The authors state: "The specific shaping rewards use for soccer are detailed in Section 4.2" but I couldn't find them there. 
---
Post-rebuttal 
My main concern was assessing the value of the overall contribution of the paper. The other reviewers seem to appreciate both the new environment being offered and the combination of techniques deployed in the authors' solution. If there is an audience that will appreciate this work at ICLR as seems to be indicated by those reviews, then I would increase my score to marginally above the acceptance threshold.