Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a selfsupervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.
# Review ICLR20, Emergent Tool Use...
This review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.
I apologize in advance for being reviewer 2.
## Overall
**Summary**
The article introduces a new multi-agent physics environment called "hide-and-seek". The authors trained agents in this environment and studied the emergence of and changes in strategies. The authors also study the performance of these same agents in new "targeted intelligence tests" compared to training from scratch and compared to agents trained with curiosity.
**Overall Opinion**
I think the environment is very appealing and the paper is overall well-structured and demonstrates novel work. Therefore I'd recommend this paper to be accepted. That being said, there are glaring issues with some of the writing that need to be addressed before I think this work conforms to the standards of ICLR. However, if these issues are addressed, I have no issue increasing my review score.
Main problems:
- The majority of the paper presents essentially a case study of what happened during a single seed of policy training. For RL literature that's very uncommon and I think it's consensus that DRL is very sensitive to random seeds. I know that you do have additional seeds in the appendix, but why didn't you mention those in the main body of the paper? You seem to have found some robustness against multiple seeds, so why not show it? And also the fact that Figure 1 & 3 only apply to 1 seed is not mentioned. I think this is easy enough to fix - I suggest since you're already at 10 pages, to just bring in the additional seeds from the appendix and average over their performance in Fig.1&3.
- The contributions section is overselling the work: (1) states that autocurricula lead to changes in agent strategy - Maybe I'm mistaken here but that sounds like a tautology. In other words, "a self-generated sequence of challenges" ("Autocurricula", according to [Leibo et al., 2019][1]) lead to changes in strategy. And (3) advertises "a proposed framework for evaluating agents in open-ended environments" and also "a suite of targeted intelligence tests for our domain". The former of those two is either not in the paper or you mean your section "6.2 Transfer and Fine-Tuning as Evaluation", which isn't novel (see e.g. [Alain & Bengio, 2016][2])
- Your acknowledgments should be anonymized until publication. Otherwise, reviewers might draw conclusions which group published this work, thus violating the double-blind review procedure.
[1]: https://arxiv.org/pdf/1903.00742.pdf
[2]: https://arxiv.org/pdf/1610.01644.pdf
Like I mentioned above, I think these are all easy to address, which should allow acceptance of this work. Here are some additional questions, comments, and nitpicks:
## Specific comments and questions
### Abstract
- "evidence that ... competition may scale better with increasing environment complexity" - that's only shown in the appendix
### Intro
- You mention TD-Gammon as a game, but I think it's an algorithm for the game Backgammon, similarly to how "Go" is the game and "AlphaGo" is an algorithm for playing.
### Rel. Work
- all good
### Hide And Seek
- Arena boundaries: What's the penalty and what's " too far outside the play area"? And in all depictions, it looks like the geometry of the arena is elevated around the edges and the agents don't have a jump action, so how would they ever go out of borders? After watching the videos: Apparently, the jagged-looking arena boundary in the videos is purely cosmetic and agents can still access that space. This is unclear from just the paper and the renderings in Figure 1.
### Policy Optimization
- Policy network and fusion are underspecified: How do you deal with the varying number of agents, boxes, obstacles? Do you just set the x/v of the missing pieces to zero or is the observation actually of a different shape in case there are more/fewer objects or agents? How's the embedding done that is depicted in Figure 2? Also, I didn't see the embedding being mentioned in the text - any reason for that?
- Figure 2 - This diagram is visually appealing but confusing and needs to be improved. Why does the agent's embedding say "1"? Why do the other agents' embeddings have a "-1" at the end of the orange box and the others don't? Is the agent's embedding concatenated with the other embeddings? If so, why and how (concat, sum, multiply, conditional batch norm, etc.)? In the center and on the right you use a blue block to indicate the agent's embedding and then at the bottom right you seem to use it as a network component or something (between the "LSTM")? If you're trying to signal that this is the agent's perception at different stages in the network, I'd use a different color to separate it from the agent's lidar and pos/vel. You don't mention that "x,v" stands for "position, velocity".
 
### Auto-curriculum and Emergent Behavior
- Figure 3: "environment-specific" (add dash). Draw skill development boundaries like in Fig.1.
- How exactly does the "surfing" work? The seekers step (not jump, right, since there is no jumping?) onto the boxes and then what? The momentum propels the box forward? Do other seekers push the box? Their movement on top of the box somehow moves the box (this seems to be the case judging by the videos but this is the least physically plausible)? This is a super interesting adaptation but I'd suspect the physics simulation to have a bug/glitch that's being exploited here.
- You mention in footnote 3 that the developmental stage and changes in reward aren't necessarily correlated. The same seems to be true for the metrics in Fig.3, which raises the question how did you come up with those boundaries for the different developmental stages in Fig.1? Did someone look at rollouts from the trained policy every couple of million steps? And do all agents learn new skills at the same time or is there a delay? From my understanding, they are all using the same policy and critic networks but maybe dependent on the proximity of an agent to an object/obstacle, it's easier or harder to execute.
### Evaluation
- clear and well-written, slightly too much content in the appendix and not enough in the main paper. Weird appendix numbering - A.6 appears in the main paper pages after A.7
### Discussion and Future Work
- all good
### Appendix
- I appreciate the TOC. I did not look into Appendix B-D because it's another 10 pages on top of the 10 pages of the article.
All in all an interesting work. Good luck with the rebuttal/discussion.