We present an artificial intelligence research platform inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a. MMOs). We demonstrate how this platform can be used to study behavior and learning in large populations of neural agents. Unlike currently popular game environments, our platform supports persistent environments, with variable number of agents, and open-ended task descriptions. The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources. Our platform aims to simulate this setting in microcosm: we conduct a series of experiments to test how large-scale multiagent competition can incentivize the development of skillful behavior. We find that population size magnifies the complexity of the behaviors that emerge and results in agents that out-compete agents trained in smaller populations.
Revised Review:
The authors of this work has taken my concerns, and concerns of other reviewers, and revised their paper during the rebuttal period. They have increased the quality of the writing / clarity, restructured the presentation (i.e. put many details in the Appendix section), and committed to open-sourcing the platform post publication. For these reasons I believe this work is now at a state that should be published at ICLR, and I revised my score from 5 to 7. I hope other reviewers can reread the work and post their updated comments.
I'm excited about the work, because it incorporates good ideas from A-Life / evolution / open-endedness communities, to introduce new paradigms and new ways of thinking to the RL community. I look forward to using this environment in my own research going forward, regardless of whether this work gets accepted or not. Good luck!
Minor comment: On page 4, the section 5 Experiments, I think "Technical details" should be in bold font before the sentence "We run each experiment using 100 worlds." so it is distinguished from being part of that sentence.
Original Review:
The authors present a new game environment inspired by MMORPGs. The environment supports a "massive" number of agents, each have different neural net brains (*) and have some foraging and combat skills. They use distributed RL to train the policies (using REINFORCE) and over time can observe the dynamics of the population of these artificial life agents interact with each other, where the only reward is survival. There are many interesting insights, such as looking at how multi-agent cooperative (and deceptive) strategies emerge, and how some agents with different niche skills co-evolve with agents with other niche skills. They also plan to open source the platform and I have high hopes that this will be a fantastic research environment. While I'm very optimistic about this work and direction, there are issues with this particular paper, and I feel it is not ready for publication in its current form. While I have no doubt that the software project will be great, as a reviewer I'm evaluating this particular paper, and I want to highlight flaws about the paper and what can be done to fix it during the rebuttal/review period.
My recommendations to improve the article:
(1) Writing - I really enjoyed this work, but frankly, the writing is horrible. It took me days of effort to decipher every paragraph and understand all the terms and what is going on. The article reads like it is written by the person who programmed the game, and played MMORPGs almost every day of his childhood and adult life, so someone who is not reading the article thru the lens of the author might have an incredibly tough time digesting the content. For instance, there are sentences like "It adds melee, ranged, and magic-based combat"... "Melee, ranged, and magic combat have maximum Manhattan distance of effect of 1, 2, and 3 cells respectively. They do 10, 2, and 1 damage respectively"... "This prevents uninteresting 'spawn killing' and is a common technique in human games". These are only a small selection of samples. There are also terminology like "#ent and #pop" which I feel should be replaced by $N_{ent}$ and $N_{pop}$ for a paper. In contrast, older works related to population-based RL training like [2], or RL in games like [3] are examples of clear and understandable writing. I highly recommend you give the draft to someone outside of your team, who is sufficiently isolated from this project (or perhaps to a professional writer if your lab has one), to go over each paragraph, and make the writing more clear. This would benefit the work in the long term as people refer back to the paper when they run your code.
(2) Diagrams - While the diagrams look interesting, IMO they are poorly made. When I look at Figure 1, 4, and 9, it is really difficult to understand what is going on. I recommend redoing the diagrams, perhaps get some inspiration from distill.pub or OpenAI blog posts. There are things that are not clear, like what the inputs are into each agent, and how the training works. I recommend having some pseudocode snippets (like the Gym framework) to explain parts of the overall picture in more detail as figures.
Given a work of this magnitude, I'm personally okay that they went over 8 pages, as long as it is properly used for clarity.
Discussion:
Concepts from Artificial Life and Evolution has been introduced in this work. There is some confusion between what is "learning" and what has been "evolved" in your setup. Some readers coming from the evolution, or biology fields (who I bet will find your paper interesting to read and experiment with) might interpret "learning" to be weight changes during a life time, while "evolution" would be changes to the weight parameters from one generation to the next, but I think in policy-gradient RL, "learning" means weight changes after an agent dies and is reborn. Should consider clarifying in the introduction, the definition of learning, and whether it is inter-life or intra-life.
You cited some of Stanley's talks on open-endedness, but I wonder if you considered their work [1] where they proposed that having a minimum criteria condition might encourage diversity of solutions. For instance, perhaps in your environment, an agent doesn't have to be the very best, but only manage to survive, to move on to the next generation, which might cause very interesting multi-agent population dynamics. A parallel to modern life is that people (at least those in wealthy nations) live with such a good social safety net that people don't really have to be the best "agent" to reproduce and survive, and this might explain the large diverse cultures and ideas we end up with as human species, compared to other animals (where the current game is probably a suitable model of). An experiment to explore an experiment where only the very weakest agents die, but leaving agents with mediocre foraging and combat skills still live on (and pursue their own interests, whatever they may be) will be super interesting, and I encourage you to explore these ideas of open-endedness.
Bugs: In the appendix, the citation for OpenAI Five needs fixing.
Currently it pains me that I can only assign a score of 5 of this work (NOTE: this has been since revised upwards to 7 upon reading revision after rebuttal period), since I don't think the current writing is up to standards. In my opinion, it deserves a score of 7-8. If you work on points (1) and (2) and submit a revised draft with much better writing, visualization, figures to explain the work, I'll happily revise my score and improve it by 1-3 points depending on how much improvement is made.
[1] Brand and Stanley. "Minimal Criterion Coevolution: A New Approach to Open-Ended Search" (GECCO 2017) http://eplex.cs.ucf.edu/papers/brant_gecco17.pdf
[2] https://arxiv.org/abs/1703.03864
[3] https://arxiv.org/abs/1804.03720
(*) well, sort of, due to compute limits they are clustered to some extent to their species, so agents within a species have identical brains, unlike the real world.