Many cooperative multiagent reinforcement learning environments provide agents with a sparse team-based reward, as well as a dense agent-specific reward that incentivizes learning basic skills. Training policies solely on the team-based reward is often difficult due to its sparsity. Also, relying solely on the agent-specific reward is sub-optimal because it usually does not capture the team coordination objective. A common approach is to use reward shaping to construct a proxy reward by combining the individual rewards. However, this requires manual tuning for each environment. We introduce Multiagent Evolutionary Reinforcement Learning (MERL), a split-level training platform that handles the two objectives separately through two optimization processes. An evolutionary algorithm maximizes the sparse team-based objective through neuroevolution on a population of teams. Concurrently, a gradient-based optimizer trains policies to only maximize the dense agent-specific rewards. The gradient-based policies are periodically added to the evolutionary population as a way of information transfer between the two optimization processes. This enables the evolutionary algorithm to use skills learned via the agent-specific rewards toward optimizing the global objective. Results demonstrate that MERL significantly outperforms state-of-the-art methods, such as MADDPG, on a number of difficult coordination benchmarks.
This paper proposes an integration of neuroevolution and gradient-based learning for reinforcement learning applications. The evolutionary algorithm focuses on sparse reward and multiagent/team optimization, while the gradient-based learning is used to inject selectively improved genotypes in the population.
This work addresses a very hot topic, i.e. the integration of NE and DRL, and the proposed method offers the positive side of both without introducing major downsides. The presented results come from a relatively simple but useful multiagent benchmark which has broad adoption. The paper is well written, presents several contributions that can be extended and ported to other work, and the results are statistically significant.
There is one notable piece missing which forces me to bridle my enthusiasm: a discussion of the genotype and of its interpretation into the network phenotype. The form taken by the actual agent is not explicitly stated; following the adoption of TD3 I would expect a policy and two critics, for a grand total of three neural networks, but this remains unverified. And if each agent is composed of three neural networks, and each individual represents a team, does this mean that each genotype is a concatenation of three (flattened) weight matrices per each agent in the team? What is the actual genotype size? It sounds huge, I would expect to be at least several hundred weights; but then this would clash with the proposed minuscule population size of 10 (recent deep neuroevolution work from Uber uses populations THREE orders of magnitude larger). Has the population size been proportionated to the genotype dimensionality? Would it be possible to reference the widely adopted defaults of industry standard CMA-ES? Speaking of algorithms, where is the chosen EA implementation discussed? The overview seems to describe a textbook genetic algorithm, but that has been overtaken as state-of-the-art since decades, constituting a poor match for TD3.
Omitting such a chapter severely limits not only the reproducibility of the work but its full understanding. For example, does the EA have sufficient population size to contribute significantly to the process, or is it just performing as a fancy version of Random Weight Guessing? Could you actually quickly run RWG with direct policy search (rather than random action selection) to establish the effective complexity of the task? My final rating after rebuttal will vary wildly depending on the ability to cover such an important piece of information. 
A few minor points, because I think that the paper appearance deserves to match the quality of the content:
- The images are consistently too small and hard to read. I understand the need to fit in the page limit by the deadline, but for the camera ready version it will be necessary to trim the text and rescale all images.
- The text is well written but often slowing down the pace for no added value, such as by dedicating a whole page to discussing a series of previously published environments.
- The hyperparameters of the evolutionary algorithm look completely unoptimized. I would expect a definite improvement in performance with minimal tuning.
- The "standard neuroevolutionary algorithm" from 2006 presented as baseline has not been state-of-the-art for over a decade. I would understand its usage as a baseline if that is indeed the underlying evolutionary setup, but otherwise I see no use for such a baseline.
-----------------------------------------------------------------------------------------------
# Update following the rebuttal phase
-----------------------------------------------------------------------------------------------
Thank you for your work and for the extended experimentation. I am confident the quality of the work is overall increased.
The core research question behind my original doubt however remains unaddressed: does the EC part of the algorithm sensibly support the gradient-descent part, or is the algorithm basically behaving as a (noisy) multi-agent TD3?
Such a contribution by itself would be undoubtedly important. Submitting it as a principled unification of EC and DL however would be more than a simple misnomer: it could mislead further research in what is an extremely promising area.
The scientific approach to clarify this point would be to design an experiment showcasing the performance of MARL using a range of sensible population sizes. To understand what "sensible" means in this context, I refer to a classic:
http://www.cmap.polytechnique.fr/~nikolaus.hansen/cec2005ipopcmaes.pdf
A lower bound for the population size with simple / unimodal fitness functions would be $4+floor(3*log(10'000)) = 31$. With such a complex, multimodal fitness though, no contribution from the EA can be expected (based on common practice in the EC field) without at least doubling or tripling that number. The upper bound does not need to be as high as with the recent Uber AI work (10k), but certainly showing the performance with a population of a few hundreds would be the minimum necessary to support your claim. A population size of 10 represents a proper lower bound for a genotype of up to 10 parameters; it is by no means within a reasonable range with your dimensionality of 10'000 parameters, and no researcher with experience in EC would expect anything but noise from such results -- with non-decreasing performance uniquely due to elitism.
The new runs in Appendice C only vary the population size for the ES algorithm, proposed as a baseline. No performance of MARL using a sensible population size is presented.
The fundamental claim is thereby unsustainable by current results. The idea is extremely intriguing and very promising, easily leading to supportive enthusiasm; it is my personal belief however that accepting this work in such a premature stage (and with an incorrect claim) could stunt further research in this direction.
[By the way, the reference Python CMA-ES implementation runs with tens of thousands of parameters and a population size of 60 in a few seconds per generation on a recent laptop: the claim of performance limitations as an excuse for not investigating a core claim suggests that more work would be better invested prior to acceptance.]