Published as a conference paper at ICLR 2018 EMERGENT COMPLEXITY VIA MULTI-AGENT COMPETITION
Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX.
In this paper, the authors produced quite cool videos showing the acquisition of highly complex skills, and they are happy about it. If you read the conclusion, this is the only message they put forward, and to me this is not a scientific message.
A more classical summary is that the authors use PPO, a state-of-the-art deep RL method, in a context where two agents are trained to perform competitive games against each other. They reuse a very recent "dense reward" technique to bootstrap the agent skills, and then anneal it to zero so that the competitive rewards obtained from defeating the opponent takes the lead. They study the effect of this annealing process (considered as a curriculum) and of various strategies for sampling the opponents. The main outcome is the acquisition of a large variety of useful skills, just observed from videos of the competitions.
The main issue with this paper is the lack of scientific analysis of the results, together with many local issues in the presentation of these results.
Below, I talk directly to the authors.
---------------------------------
The related work subsection is just a list of works, it should explain how the proposed work position itself with respect to these works.
In Section 5.2, you are just describing "cool" behaviors observed from your videos.
Science is about producing quantitative results, analyzing them and discussing them.
I would be glad to read more science about these cool behaviors. Can you define a repertoire of such behaviors?
Determine how often they are discovered? Study how the are represented in the networks?
Anything beyond "look, that's great!" would make the paper better...
By the end of Section 5.2, you allude to transfer learning phenomena.
It would be nice to study these transfer effects in your results with a quantitative methodology.
Section 5.3 is more scientific, but it has serious issues.
In all subfigures in Figure 3, the performance of opponents should be symmetric around 50%. This is not the case for subfigures (b) and (c-1). Why?
Do they correspond to non-zero sum game? The x-label is "version". Don't you mean "number of epochs", or something like this? Why do the last 2 images
share the same caption?
I had a hard time understanding the message from Table 1. It really needs a line before the last row and a more explicative caption.
Still in 5.3, "These results echo"...: can you characterize this echo? What is the relationship to this other work?
Again, "These results shed further light": further with respect to what? Can you be more explicit about what we learn?
Also, I find that annealing a kind of reward with respect to another is a weak form of curriculum learning. This should be further discussed.
In Section 5.4, the idea of using many opponents from many stages of learning in not new.
If I'm correct, the same was done in evolutionary method to escape the "arms race" dead-end in prey-predator races quite a while ago  (see e.g. "Coevolving predator and prey robots: Do “arms races” arise in artificial evolution?" Nolfi and Floreano, 1998)
Section 5.5.1 would deserve a more quantitative presentation of the effect of randomization.
Actually, in Fig5: the axes are not labelled. I don't believe it shows a win-rate. So probably the caption (or the image) is wrong.
In Section 5.5.2, you "suspect this is because...".
The role of a scientific paper is to clearly establish results and explanation from solid quantitative analysis. 
-------------------------------------------
More local comments:
Abstract:
"Normally, the complexity of the trained agent is closely related to the complexity of the environment." Here you could cite Herbert Simon (1962).
"In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself."
Well, for an agent, the other agent(s) are part of its environment, aren't they? So I don't like this perspective that the environment itself is "simple".
Intro:
"RL is exciting because good RL exists." I don't believe this is a strong argument. There are many good things that exist which are not exciting.
"In general, training an agent to perform a highly complex task requires a highly complex environment, and these can be difficult to create." Well, the standard perspective is the other way round: in general, you face a complex problem, then you need to design a complex agent to solve it, and this is difficult. 
"This happens because no matter how weak or strong an agent is, an environment populated with other agents of comparable strength provides the right challenge to the agent, facilitating maximally rapid learning and avoiding getting stuck." This is not always true. The literature is full of examples where two-players competition end-up with oscillations between to solutions rather than ever-increasing skill performance. See the prey-predator literature pointed above.
"in the domain of continuous control, where balance, dexterity, and manipulation are the key skills." In robotics, dexterity, and manipulation usually refer to using the robot's hand(s), a capability which is not shown here.
In preliminaries, notation, what you describe corresponds to the framework of Dec-POMDPs, you should position yourself with respect to this framework (see e.g. Memory-Bounded Dynamic Programming for DEC-POMDPs. S Seuken, S Zilberstein)
In PPO description : Let l_t(\theta) ... denote the likelihood ratio: of what?
p5:
would train on the dense reward for about 10-15% of the trainig epochs. So how much is \alpha_t? How did you tune it? Was it hard?
p6:
you give to the agent the mass: does the mass change over time???
In observations: Are both agents given different observations? Could you specify which is given what?
In Algorithms parameters: why do you have to anneal longer for kick-and-defend? What is the underlying phenomenon?
In Section 5, the text mentions Fig5 before Fig4.
-------------------------------------------------
Typos:
p4:
research(Andrychowicz => missing space
straight forward => straightforward
p5:
agent like humanoid(s)
from exi(s)ting work
p6:
eq. 1 => Eq. (1) (you should use \eqref{})
In section 4.1 => In Section 4.1 (same p7 for Section 4.2)
"One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore the benefit of this exploration reward. As already argued, we found the exploration reward to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward." => One question that arises is the extent to which the outcome of learning is affected by this exploration reward and to explore its benefit. As already argued, we found it to be crucial for learning as otherwise the agents are unable to explore the sparse competition reward.
p8:
in a local minima => minimum
p9:
in references, you have Jakob Foerster and Jakob N Foerster => try to be more consistent.
p10, In Laetitia Matignon et al.  ... markov => Markov
p11, I would rename C_{alive} as C_{standing}