summaries: This paper proposes to employ conditional variational autoencoder (CVAE) to estimate the geometry of 2D rotations of objects given images partially labeled. Here, the label represents the geometry of the 2D rotation. The proposed method introduces two latent representation. z is the ordinal latent variable and r a latent representation for the rotations where the latent variable is defined in the 1-dimensional circle in R^2 so that it can naturally represent a hyperspherical latent space.

summaries: This paper presents a semi-supervised approach to learn the rotation of objects in an image. The primary motivation is that for rotation estimation datasets may not always be fully labeled, so learning partially from labeled and partially for unlabeled is important. The approach is to use a CVAE with a supervised loss and an unsupervised loss and to jointly train the network. Limited experiments that show performance are presented.

abstractText: Pose estimation is the task of finding the orientation of an object within an image with respect to a fixed frame of reference. Current classification and regression approaches to the task require large quantities of labelled data for their purposes. The amount of labelled data for pose estimation is relatively limited. With this in mind, we propose the use of Conditional Variational Autoencoders (CVAEs) Kingma et al. (2014) with circular latent representations to estimate the corresponding 2D rotations of an object. The method is capable of training with datasets that have an arbitrary amount of labelled images providing relatively similar performance for cases in which 10-20% of the labels for images is missing.

authors:
Markus Braun
Qing Rao
Yikang Wang
Fabian Flohr

title: Pose-RCNN: Joint object detection and pose estimation using 3d object proposals

venue: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC,

authors:
Taco S Cohen
Mario Geiger
Jonas Koehler
Max Welling

title: Spherical CNNs

authors:
Tim R. Davidson
Luca Falorsi
Nicola De Cao
Thomas Kipf
Jakub M. Tomczak

title: Hyperspherical Variational Auto-Encoders

authors:
Carlos Esteves
Christine Allen-Blanchette
Ameesh Makadia
Kostas Daniilidis

title: Learning SO(3) Equivariant Representations with Spherical CNNs. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

authors:
Carlos Esteves
Avneesh Sud
Zhengyi Luo
Kostas Daniilidis
Ameesh Makadia

title: Cross-Domain 3D Equivariant Image Embeddings

authors:
Luca Falorsi
Pim de Haan
Tim R. Davidson
Nicola De Cao
Maurice Weiler
Patrick Forré
Taco S. Cohen

title: Explorations in Homeomorphic Variational Auto-Encoding

venue: URL http: //arxiv.org/abs/1807.04689

authors:
Matthew Johnson-Roberson
Charles Barto
Rounak Mehta
Sharath Nittur Sridhar
Karl Rosaen
Ram Vasudevan

title: Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks

venue: Proceedings - IEEE International Conference on Robotics and Automation,

authors:
Diederik P Kingma
Max Welling

title: Auto-Encoding Variational Bayes

venue: In International Conference on Learning Representations (ICLR),

authors:
Diederik P. Kingma
Danilo J. Rezende
Shakir Mohamed
Max Welling

title: Semi-Supervised Learning with Deep

venue: Generative Models. pp

authors:
Tejas D. Kulkarni
Will Whitney
Pushmeet Kohli
Joshua B. Tenenbaum

title: Deep Convolutional Inverse Graphics Network

authors:
Henry Li
Ofir Lindenbaum
Xiuyuan Cheng
Alexander Cloninger

title: Variational Random Walk Autoencoders

authors:
Li-Jia Li
Kai Li
Fei Fei Li
Jia Deng
Wei Dong
Richard Socher
Li Fei-Fei

title: ImageNet: a Large-Scale Hierarchical Image Database Human trajectory forecasting View project Shrimp Project View project ImageNet: A Large-Scale Hierarchical Image Database

venue: IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Zhigang Li
Yuwang Wang
Xiangyang Ji

title: Monocular Viewpoints Estimation for Generic Objects in the Wild

venue: IEEE Access,

authors:
Shuai Liao
Efstratios Gavves
Cees G.M. Snoek

title: Spherical Regression: Learning Viewpoints

venue: Surface Normals and 3D Rotations on n-Spheres

authors:
Siddharth Mahendran
Haider Ali
Rene Vidal

title: 3D Pose Regression Using Convolutional Neural Networks

venue: Proceedings - 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017,

title: A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Images

authors:
Eric Marchand
Hideaki Uchiyama
Fabien Spindler

title: Pose Estimation for Augmented Reality: A Hands-On Survey

venue: IEEE Transactions on Visualization and Computer Graphics,

authors:
Emile Mathieu
Charline Le Lan
Chris J. Maddison
Ryota Tomioka
Yee Whye Teh

title: Hierarchical Representations with Poincaré Variational Auto-Encoders

authors:
Luis A. Pérez Rey
Vlado Menkovski
Jacobus W. Portegies

title: Diffusion Variational Autoencoders

authors:
Sergey Prokudin
Peter Gehler
Sebastian Nowozin

title: Deep Directional Statistics: Pose Estimation with Uncertainty Quantification

authors:
Helge Rhodin
Mathieu Salzmann
Pascal Fua

title: Unsupervised geometry-aware representation for 3D human pose estimation

venue: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),

authors:
Nima Sedaghat
Thomas Brox

title: Unsupervised Generation of a View Point Annotated Car Dataset from Videos

authors:
Hao Su
Charles R. Qi
Yangyan Li
Leonidas J. Guibas

title: Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views

venue: Proceedings of the IEEE International Conference on Computer Vision, 2015 Inter:2686–2694,

authors:
Shubham Tulsiani
Jitendra Malik

title: Viewpoints and keypoints

venue: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,

authors:
Gul Varol
Javier Romero
Xavier Martin
Naureen Mahmood
Michael J. Black
Ivan Laptev
Cordelia Schmid

title: Learning from Synthetic Humans

venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2017-Janua, pp. 4627–4635. IEEE,

authors:
Zhirong Wu
Shuran Song
Aditya Khosla
Fisher Yu
Linguang Zhang
Xiaoou Tang
Jianxiong Xiao

title: 3D ShapeNets: A Deep Representation for Volumetric Shapes

authors:
Yu Xiang
Roozbeh Mottaghi
Silvio Savarese

title: Beyond PASCAL: A benchmark for 3D object detection in the wild

venue: IEEE Winter Conference on Applications of Computer Vision,

authors:
Yu Xiang
Tanner Schmidt
Venkatraman Narayanan
Dieter Fox

title: PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. 2018

heading: 1 INTRODUCTION

text:
Pose estimation is the computer vision task in which the rotation of an object is estimated with respect to a certain frame of reference. The estimation of the position and the pose of an object within an image is an important challenge of computer vision and can be applied to a wide range of areas such as car navigation (Braun et al., 2016), augmented reality (Marchand et al., 2016) and robot interaction (Xiang et al., 2018).
Several methods based on the use of convolutional neural networks have been proposed high performance. These approaches are divided into two: classification-based, where the possible rotations that can be predicted are discretized into predefined sets with different coarseness levels and regression-based where a continuous prediction is provided.
Most of them are supervised methods which require large datasets with accurately labelled images and with a sufficiently good coverage of all possible rotations of each object instance. One of such datasets obtained from natural images and manually annotating objects in images is the Pascal3D+ dataset (Xiang et al., 2014) with about 22 thousand images from twelve categories. Compared to other datasets for other computer vision tasks such as classification (with millions of images) Li et al. (2009) the example data is scarce and the labelling process might be cumbersome and prone to errors. Thus, the need for a method to train with a limited amount of labelled data.
To solve this problem there have been dataset generation approaches proposed to create new training labels and images (Su et al., 2015; Varol et al., 2017; Johnson-Roberson et al., 2017) from 3D models and inserting them in natural scenery. Other approaches are focused on reducing the supervision to auxiliary tasks indirectly related to the pose estimation such as in Esteves et al. (2018b).Prokudin et al. (2018) addresses the problem of label quality by including an uncertainty estimation in the rotation predictions.
The work presented in this paper proposes a semi-supervised probabilistic method for pose estimation that incorporates the geometry of 2D rotations into the latent variable representations. The method is capable of using both labelled and unlabelled images from an object to improve the rotation regressions by performing the auxiliary task of reconstruction from the autoencoders. We have also investigated the effects of having a variable amount of labelled data in a controlled setting and variable coverage of the rotations.
In the general setting of object pose estimation an object is located in the scene of interest and an image has been taken of it. The object has a certain rotation with respect to the a fixed frame of reference. The goal is to estimate the rotation that the object has undergone from the available
image. In particular we are interested in the case were we might have a dataset in which the objects might be partially labelled.
More formally, consider that we have a training dataset consisting of L labelled images with their corresponding rotation representations {(xl, rl)}Ll=1 in which xl ∈ X and rl ∈ R correspond to the l-th image and rotation, respectively, and U unlabelled images {xu}Uu=1. How can we obtain a model for estimating the pose of an object based on an image, taking advantage of both datasets? How does such a model perform with respect to the amount of supervision provided and the angle coverage?

heading: 2 RELATED WORK

text:
Current approaches for rotation estimation using neural networks can be divided into classification and regression. Classification methods discretize the possible rotations orientations of an object (Su et al., 2015; Li et al., 2019b; Tulsiani & Malik, 2015) while regression methods provide a more approach to the continuous nature of rotations (Mahendran et al., 2018a;b; Liao et al., 2019; Prokudin et al., 2018).
Since rotations have a particular topology, their structure needs to be taken into account for both classification and regression tasks. Mahendran et al. (2018a) studies the performance of regression methods by using different rotation representations in a supervised setting. Liao et al. (2019) focus on the task of regression by representing rotations in n-spheres for different pose-related tasks. Prokudin et al. (2018).
In Esteves et al. (2018b;a) equivariant representations are used which capture the group structure of rotations. Their method is trained with the aid of spherical convolutional neural networks (Cohen et al., 2018) based on an auxiliary classification task, this method is unsupervised in the sense that it does not need labels for angles and is capable of estimating relative rotation among different views of an object.
These supervised methods still require the use of labelled data in good quantity and quality. To circumvent these needs, we are interested in the semi-supervised and unsupervised approaches such as Esteves et al. (2018b;a); Rhodin et al. (2018); Kulkarni et al. (2015). Similar to Kulkarni et al. (2015), our focus is to use probabilistic models to capture the topological structure of rotations.
In particular, within the Variational Autoencoders (VAE) Kingma & Welling (2014) context there is a recent growing interest in the study of latent variables with different geometries: hyperspherical (Davidson et al., 2018), SO(3) (Falorsi et al., 2018), hyperbolic (Mathieu et al., 2019) and arbitrary closed manifolds (Pérez Rey et al., 2019; Li et al., 2019a).
VAEs provide an unsupervised method for latent variable estimation and in principle could be used for capturing the underlying geometrical features for a dataset. The Conditional Variational Autoencoder (CVAE) (Kingma et al., 2014) allows to introduce supervision into the latent variable representations.Prokudin et al. (2018) used the CVAE as a supervised method for pose prediction and adding an uncertainty estimation for the input data. This method could not be directly extended to semi-supervised pose estimation without extra modifications.
We use the previous work as a starting point for developing a semi-supervised method for pose estimation based on CVAEs with underlying geometrical latent variables for pose estimation.

heading: 3 METHODS

text:
ROTATION REPRESENTATION
It is important to choose an appropriate latent representation for the rotations that captures their topological properties. This has been discussed in Liao et al. (2019) and Mahendran et al. (2018a) where several representations are described. In the case of 2D planar rotations along one fixed axis each rotation can be described in terms of the 1-dimensional circle S1 Liao et al. (2019). Our method will has focus on the use of a probabilistic models (Variational Autoencoders Kingma & Welling (2014) capable of encoding the input images in a circular latent space.
To the best of our knowledge, there are three approaches to implement a variational autoencoder with a hyperspherical latent space. Firstly in Davidson et al. (2018), they have focused on a method that is tailored specifically for hypersphere. On the other hand, Pérez Rey et al. (2019) and Li et al. (2019a) propose methods for choosing arbitrary closed spaces. We have chosen to focus on using the work in Pérez Rey et al. (2019) to represent the rotations in S1 since it can provide arbitrary manifolds as latent space which can be used in further experiments on different applications.
CONDITIONAL VARIATIONAL AUTOENCODERS
The Conditional Variational Autoencoders (CVAE) introduced by Kingma et al. (2014) consist of a method for approximating to the true generative process that produced a dataset in a semi-supervised way. The CVAE assumes a generative process in which the data spaceX and two latent spacesR,Z are involved. The data space X consists in this case of the set of all possible images with a certain width height and color channels. The latent space R is associated with the variable of interest that we have available in the labelled dataset (in this case the rotation), and the second latent space Z corresponds to an unobserved variable.
In the Variational Autoencoder (VAE) context (Kingma & Welling, 2014), a family of parametric distributions is used to approximate the posterior distributions over the latent spaces. The distribution over the rotations R is denoted by QR|x and the distribution over the latent space Z is denoted by QZ|x,r. Such distributions are denominated encoding distributions and their parameters are calculated with the aid of neural networks. On the other hand a parametric distribution over the data space X is denominated decoding distribution and denoted by PX|z,r.
The generative model is obtained via variational inference by optimizing the parameters of the encoding and decoding distribution in order to maximize the evidence lower bound of the loglikelihood of the available observations (images and rotations). For the labelled dataset the loglikelihood of the image-rotation pairs is maximized. The evidence lower bound for the labelled dataset L(x, r) ≤ logPX×R(x, r) for a pair of image and rotation (x, r) ∈ X ×R is given by
L(x, r) = Ez∼QZ|x,r [ logPX|z,r(x) + log (PR(r)) + log (PZ(z))− log (QZ|x,r(z)) ] . (1)
Where PR and PZ are the chosen prior probability density functions of the prior distributions over the corresponding latent spaces. In the case of the unlabelled dataset the evidence lower bound U(x) ≤ logPX(x) takes the form
U(x) = E(z,r)∼QZ×R|x [ logPX|z,r(x) + log (PR(r)) + log (PZ(z))− log (QZ×R|x(z, r)) ] . (2)
For our experiments, the latent space Z is chosen as a d-dimensional Euclidean space Z = Rd with a normal posterior approximate with diagonal covariance Q(µZ ,σZ)Z|x,r and a standard normal distribution as prior PZ where µZ is the location parameter and σZ is a vector representing the diagonal of the covariance matrix.
The chosen latent space R that captures the 2D rotations of the object is a circle embedded in R2 i.e. R = S1 ⊆ R2. The encoding distribution Q(µR,tR)R|x over R is chosen as in Pérez Rey et al. (2019) where it corresponds to the transition kernel of the diffusion equation over the circle such that µR and tR correspond to the location and scale parameters respectively. A uniform prior PR overR is selected over this latent space. For the decoding distribution P(pX)X|z,r we propose a Bernoulli distribution over each of the pixels with parameter pX .
According to Kingma et al. (2014) an extra regression loss term is added to the CVAE objective. This loss term can be considered as the regression loss associated to the posterior over R that estimates the rotations. For an input image xl and its corresponding true rotation rl ∈ S1, the regression loss corresponds to the negative dot product between the estimated location of the posterior approximate µR(xl) and the true rotation i.e.
S(rl, xl) = −rl · µR(xl). (3)
The maximization of the evidence lower bounds for the unlabelled and labelled dataset can be translated into a minimization problem by introducing a minus sign. The minimization loss for the labelled and unlabelled datasets is then described by the following formula
J ( {xl, rl}Ll=1, {xu}Uu=1 ) = − L∑ l=1 (L(xl, rl)− S(rl, xl))− U∑ u=1 U(xu). (4)
TRAINING THE CVAE
In order to train a CVAE the labelled and unlabelled data need to be fed to the encoding and decoding distributions respectively. Figure 1 shows how the parameters of the corresponding encoding and decoding distributions are estimated for the labelled and the unlabelled datasets. The sampling from Q(µZ ,σZ)Z|xl,rl is done using the reparametrization trick from Kingma & Welling (2014) while the sampling from Q(µR,tR)R|xu is done by performing a random walk over the circle as described in Pérez Rey et al. (2019).
In order to train jointly the whole dataset ofN = L+U unlabelled and labelled images we introduce for each available image xi an additional supervision value si ∈ {0, 1}. If the datapoint is labelled then si = 1 and si = 0 otherwise. For unlabelled images xi an abritrary fixed label yi is provided. During training the loss function to be optimized corresponds to
J ( {xi, ri, si}Ni=1 ) = −si N∑ l=1 (L(xi, ri)− S(ri, xi))− (1− si) N∑ i=1 U(xi). (5)
This loss function reduces to Equation 4 but provides a practical approach for implementing the training of labelled and unlabelled data.

heading: 4 DATASET GENERATION AND ARCHITECTURE

text:
In order to test the capabilities of our method we have restricted the use of our method to a controlled setting in which the data annotations and the images provided can be generated arbitrarily. With this purpose we have focused on the use of a dataset of 3D models from which we can generate arbitrary amount of rendered images of objects with known rotations.
ModelNet40 by Wu et al. (2014) is a dataset of 3D CAD models with object instances from forty categories. For the task of pose estimation, an aligned version was provided by Sedaghat & Brox (2015) in which, for each category, all object instances were aligned to a certain frame of reference. From the 3D models we have focused on the airplane class which has the most training instances (626 instances) and used a regular rendering procedures to generate the images for training and testing the CVAE:In the case of the planar 2D rotations of the object, we have generated the views by rendering images from 64 equally distanced angles within the interval [0, 2π].
We train with a VGG-like architecture for both the encoder and decoder neural networks similar to the work presented in Prokudin et al. (2018). The encoder networks, for the parameters of QR|x and QZ|x,r, consist of six convolutional layers with 24, 48, 48, 64 and 64 feature channels followed by a fully connected layer of 512 neurons. Each convolution and dense layer has rectifying linear unit activations followed by batch normalization. A final dense layer with linear activations is attached to each network to predict the pertaining parameters to be estimated. To enforce the restriction of positive scale parameters the log-scale parameters are estimated for each encoding density instead.
For the latent space Z = Rd the dimension is chosen as d = 50 with location and log-scale parameters of the same dimension. For the space of rotations S1 ⊆ R2 the location parameter is obtained from a 2 neuron dense layer followed by Euclidean normalization. The log-scale parameter is the output of a 1 neuron dense layer.
The decoding neural network consists of the reverse VGG network. First a dense layer with 512 neurons is followed by a combination of up-sampling and convolutional layers with filters 64, 64, 48, 48, 24. The final convolution has a sigmoid activation and returns arrays with same shape as input images.

heading: 5 EXPERIMENTS

text:
In our experiments we have studied the performance of our approach with respect to two main factors. The amount of labelled data and the coverage of the rotations that are fed to the network.
Number of renders As it was described in the previous section, for each object instance 64 images were generated at different rotations. During training, different subsets of these renders were used to test the effects of the angle coverage. The number of renders used per object instance were {64, 32, 16, 8}.
Label percentage For testing the effects of changing the amount of available labels we select uniformly randomly a certain percentage of object instances for which we remove all the labels in their corresponding images. We further divide randomly the available labelled images into train(70%) and validation (30%). In each experiment we train for 200 epochs with early stopping and a tolerance of 15 epochs for the loss. The model with the lowest validation loss is chosen as the final model for the given experiment.
EVALUATION
Evaluation of the rotation regression is done with respect to the rendered images from object instances within the test set of ModelNet40 aligned. For each 3D model instance a set of 64 images at angles equally separated from the interval [0, 2π] were generated. All of the trained models are evaluated with respect to the complete test set of 64 images per object instance. For each experiment three repetitions are trained and the relevant metrics are averaged and their standard deviation is reported.
Median Error We consider that the location parameter obtained for an image in the test dataset µR(xl) ∈ R = S1 corresponds to the rotation prediction for the labelled image xl in the test dataset. The angular error between the label rotation rl ∈ S1 and the prediction µR(xl) is calculated as their geodesic distance in S1
dS1(rl, µR(xl)) = arccos(rl · µR(xl)). (6)
We calculate the median error across the images in the test dataset and report it in degrees.
Accuracy For accuracy, the prediction is considered to be correct if its geodesic distance is smaller than a certain threshold θ, i.e. dS1(rl, µR(xl)) ≤ θ. We consider three threshold values θ ∈ {π6 , π 12 , π 24} for our evaluation and report the percentages of correct classifications.

heading: 6 RESULTS

text:
The results from Figure 2 show the performance of the models with respect to the amount of supervision. A general trend can be distinguished in all plots i.e. increasing the amount of labelled images decreases the median error and thus improves the models’ performance. This is an expected result since the models receive more signals that push the training into the correct regression values. It is important to notice that when a relatively small amount of labels is missing (around 80%-90%) the method achieves a similar performance. This means that the method can be used effectively in cases with a moderately restricted labelling of data.
Focusing on the results with respect to the different number of renders per object instance and evaluated with respect to the test images from, it is expected to observe higher variance in the methods trained on less images. For such models, the training has been biased towards a restricted set of views which increases the variability in the predictions for the test set.
It is quite interesting to find that even though the models trained on 8 and 16 renders have seen a restricted amount of views from the object instances, their performance is high when trained on highly labelled data. This means that under high supervision, the amount of example views from an object at different angles can be decreased significantly while maintining competitive performances.
Table 1 shows a comparison of the performance of the method in the fully supervised setting. In such cases the CVAE has a good performance with the median error decreasing with respect to the number of renders provided during training as expected. As it was previously described, the results show that even with few representative samples of an object’s rotation (with 8 renders), the method still achieves a low median error and high accuracy (up to 87 ± 16.3%). This performance still comes at the cost of high variance in the resulting models’ performance.

heading: 7 CONCLUSION

text: We have presented a novel approach to the task of semi-supervised pose estimation with limited trained data. The method is capable of receiving a dataset of labelled and unlabelled images. As expected, model’s performance increases with the amount of supervision provided to the network. Moreover, when just a small percentage of the training labels are missing the model can achieve comparable performance to that of the fully supervised method. Some future steps to this work include the use of different sampling schemes for generated the rendered views of an object. New sampling schemes would also include arbitrary 3D rotations. Moreover, testing with more challenging datasets such as Pascal3D+ Xiang et al. (2014) would provide a point of comparison with other pose estimation methods..


summaries: The paper proposes a kind of curriculum for large-scale multi-agent learning. The related work section mentions some obvious points of comparison (note: see also https://science.sciencemag.org/content/364/6443/859.abstract). However, the authors do not compare with ANY of this work (either in terms of algorithm design or performance). It is therefore difficult to evaluate the contribution.

summaries: This paper proposes a new method of scaling multi-agent reinforcement learning to a larger number of agents using evolution. Specifically, the procedures (EPC) involves starting with a small number of agents, training multiple sets in parallel, and doing crossover to find the set of agents that generalize best to a larger number of agents. This is motivated by the intuition that agents that perform best in small groups may not be the ones that perform best in larger groups. These claims are empirically verified in three games based on the particle world set of environments.

abstractText: In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up MultiAgent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially. The source code and videos can be found at https://sites.google.com/view/epciclr2020/.

authors:
Thomas Bäck
Hans-Paul Schwefel

title: An overview of evolutionary algorithms for parameter optimization

venue: Evolutionary computation,

authors:
Bowen Baker
Ingmar Kanitscheider
Todor Markov
Yi Wu
Glenn Powell
Bob McGrew
Igor Mordatch

title: Emergent tool use from multi-agent autocurricula

authors:
Trapit Bansal
Jakub Pachocki
Szymon Sidor
Ilya Sutskever
Igor Mordatch

title: Emergent complexity via multi-agent competition

venue: In ICLR,

authors:
Yoshua Bengio
Jérôme Louradour
Ronan Collobert
Jason Weston

title: Curriculum learning

venue: In ICML, pp

authors:
Edoardo Conti
Vashisht Madhavan
Felipe Petroski Such
Joel Lehman
Kenneth Stanley
Jeff Clune

title: Improving exploration in evolution strategies for deep reinforcement learning via a population of noveltyseeking agents

venue: In NeurIPS,

authors:
Wojciech Marian Czarnecki
Siddhant M Jayakumar
Max Jaderberg
Leonard Hasenclever
Yee Whye Teh
Nicolas Heess
Simon Osindero
Razvan Pascanu

title: Mix & match agent curricula for reinforcement learning

venue: In ICML,

authors:
Yan Duan
Marcin Andrychowicz
Bradly Stadie
Jonathan Ho
Jonas Schneider
Ilya Sutskever
Pieter Abbeel
Wojciech Zaremba

title: One-shot imitation learning

venue: In NIPS,

title: Learning and development in neural networks: The importance of starting small

authors:
Carlos Florensa
David Held
Markus Wulfmeier
Michael Zhang
Pieter Abbeel

title: Reverse curriculum generation for reinforcement learning

venue: In CoRL,

authors:
Jakob Foerster
Ioannis Alexandros Assael
Nando de Freitas
Shimon Whiteson

title: Learning to communicate with deep multi-agent reinforcement learning

authors:
Jakob N Foerster
Gregory Farquhar
Triantafyllos Afouras
Nantas Nardelli
Shimon Whiteson

title: Counterfactual multi-agent policy gradients

venue: In AAAI,

authors:
Tanmay Gangwani
Jian Peng

title: Genetic policy optimization

authors:
Matthew Hausknecht
Peter Stone

title: Deep reinforcement learning in parameterized action space

venue: arXiv preprint arXiv:1511.04143,

authors:
He He
Jordan Boyd-Graber
Kevin Kwok
Hal Daumé III

title: Opponent modeling in deep reinforcement learning

venue: In International Conference on Machine Learning,

authors:
Rein Houthooft
Yuhua Chen
Phillip Isola
Bradly Stadie
Filip Wolski
OpenAI Jonathan Ho
Pieter Abbeel

title: Evolved policy gradients

authors:
Shariq Iqbal
Fei Sha

title: Actor-attention-critic for multi-agent reinforcement learning

authors:
Max Jaderberg
Valentin Dalibard
Simon Osindero
Wojciech M Czarnecki
Jeff Donahue
Ali Razavi
Oriol Vinyals
Tim Green
Iain Dunning
Karen Simonyan

title: Population based training of neural networks

venue: arXiv preprint arXiv:1711.09846,

authors:
Max Jaderberg
Wojciech M Czarnecki
Iain Dunning
Luke Marris
Guy Lever
Antonio Garcia Castaneda
Charles Beattie
Neil C Rabinowitz
Ari S Morcos
Avraham Ruderman

title: Human-level performance in first-person multiplayer games with population-based deep reinforcement learning

venue: arXiv preprint arXiv:1807.01281,

authors:
Jiechuan Jiang
Zongqing Lu

title: Learning attentional communication for multi-agent cooperation

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

venue: arXiv preprint arXiv:1710.10196,

authors:
Shauharda Khadka
Kagan Tumer

title: Evolution-guided policy gradient in reinforcement learning

authors:
Sergey Levine
Chelsea Finn
Trevor Darrell
Pieter Abbeel

title: End-to-end training of deep visuomotor policies

venue: The Journal of Machine Learning Research,

authors:
Kaixiang Lin
Renyu Zhao
Zhe Xu
Jiayu Zhou

title: Efficient large-scale fleet management via multi-agent deep reinforcement learning

venue: In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,

title: Markov games as a framework for multi-agent reinforcement learning

authors:
Siqi Liu
Guy Lever
Nicholas Heess
Josh Merel
Saran Tunyasuvunakool
Thore Graepel

title: Emergent coordination through competition

authors:
Ryan Lowe
Yi Wu
Aviv Tamar
Jean Harb
Pieter Abbeel
Igor Mordatch

title: Multi-agent actor-critic for mixed cooperative-competitive environments

authors:
Aleksandra Malysheva
Tegg Taekyong Sung
Chae-Bong Sohn
Daniel Kudenko
Aleksei Shpilman

title: Deep multi-agent reinforcement learning with relevance graphs

venue: arXiv preprint arXiv:1811.12557,

title: Digital humanitarians: how big data is changing the face of humanitarian response

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Alex Graves
Ioannis Antonoglou
Daan Wierstra
Martin Riedmiller

title: Playing atari with deep reinforcement learning

venue: arXiv preprint arXiv:1312.5602,

authors:
Igor Mordatch
Pieter Abbeel

title: Emergence of grounded compositional language in multi-agent populations

authors:
Adithyavairavan Murali
Lerrel Pinto
Dhiraj Gandhi
Abhinav Gupta

title: Cassl: Curriculum accelerated self-supervised learning

venue: In ICRA,

authors:
Liviu Panait
Sean Luke

title: Cooperative multi-agent learning: The state of the art

authors:
Peng Peng
Quan Yuan
Ying Wen
Yaodong Yang
Zhenkun Tang
Haitao Long
Jun Wang

title: Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games

venue: arXiv preprint arXiv:1703.10069,

authors:
Tim Salimans
Jonathan Ho
Xi Chen
Szymon Sidor
Ilya Sutskever

title: Evolution strategies as a scalable alternative to reinforcement learning

venue: arXiv preprint arXiv:1703.03864,

authors:
Yoav Shoham
Rob Powers
Trond Grenager

title: Multi-agent reinforcement learning: a critical survey

venue: Web manuscript,

authors:
David Silver
Guy Lever
Nicolas Heess
Thomas Degris
Daan Wierstra
Martin Riedmiller

title: Deterministic policy gradient algorithms

authors:
Joseph Suarez
Yilun Du
Phillip Isola
Igor Mordatch

title: Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents

authors:
Sainbayar Sukhbaatar
Rob Fergus

title: Learning multiagent communication with backpropagation

authors:
Sainbayar Sukhbaatar
Zeming Lin
Ilya Kostrikov
Gabriel Synnaeve
Arthur Szlam
Rob Fergus

title: Intrinsic motivation and automatic curricula via asymmetric self-play

venue: arXiv preprint arXiv:1703.05407,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Rui Wang
Joel Lehman
Jeff Clune
Kenneth O Stanley

title: Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions

authors:
Xiaolong Wang
Ross Girshick
Abhinav Gupta
Kaiming He

title: Non-local neural networks

venue: In CVPR,

authors:
Michael P Wellman
Daniel M Reeves
Kevin M Lochner
Shih-Fen Cheng
Rahul Suri

title: Approximate strategic reasoning through hierarchical reduction of large symmetric games

authors:
Mason Wright
Yongzhao Wang
Michael P Wellman

title: Iterated deep reinforcement learning in games: history-aware training for improved stability

venue: In Proceedings of the 2019 ACM Conference on Economics and Computation,

authors:
Cathy Wu
Aboudy Kreidieh
Eugene Vinitsky
Alexandre M Bayen

title: Emergent behaviors in mixed-autonomy traffic

authors:
Yi Wu
Yuxin Wu
Georgia Gkioxari
Yuandong Tian

title: Building generalizable agents with a realistic and rich 3d environment

venue: arXiv preprint arXiv:1801.02209,

authors:
Yuxin Wu
Yuandong Tian

title: Training agent for first-person shooter game with actor-critic curriculum learning

authors:
Wei Yang
Xiaolong Wang
Ali Farhadi
Abhinav Gupta
Roozbeh Mottaghi

title: Visual semantic navigation using scene priors

authors:
Yaodong Yang
Rui Luo
Minne Li
Ming Zhou
Weinan Zhang
Jun Wang

title: Mean field multi-agent reinforcement learning

authors:
Vinicius Zambaldi
David Raposo
Adam Santoro
Victor Bapst
Yujia Li
Igor Babuschkin
Karl Tuyls
David Reichert
Timothy Lillicrap
Edward Lockhart

title: Relational deep reinforcement learning

venue: arXiv preprint arXiv:1806.01830,

authors:
Yuke Zhu
Roozbeh Mottaghi
Eric Kolve
Joseph J Lim
Abhinav Gupta
Li Fei-Fei
Ali Farhadi

title: Targetdriven visual navigation in indoor scenes using deep reinforcement learning

heading: 1 INTRODUCTION

text:
Most real-world problems involve interactions between multiple agents and the problem becomes significantly harder when there exist complex cooperation and competition among agents. Inspired by the tremendous success of deep reinforcement learning (RL) in single-agent applications, such as Atari games (Mnih et al., 2013), robotics manipulation (Levine et al., 2016), and navigation (Zhu et al., 2017; Wu et al., 2018; Yang et al., 2019), it has become a popular trend to apply deep RL techniques into multi-agent applications, including communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Mordatch & Abbeel, 2018), traffic light control (Wu et al., 2017), physical combats (Bansal et al., 2018), and video games (Liu et al., 2019; OpenAI, 2018).
A fundamental challenge for multi-agent reinforcement learning (MARL) is that, as the number of agents increases, the problem becomes significantly more complex and the variance of policy gradients can grow exponentially (Lowe et al., 2017). Despite the advances on tackling this challenge via actor-critic methods (Lowe et al., 2017; Foerster et al., 2018), which utilize decentralized actors and centralized critics to stabilize training, recent works still scale poorly and are mostly restricted to less than a dozen agents. However, many real-world applications involve a moderately large population of agents, such as algorithmic trading (Wellman et al., 2005), sport team competition (Hausknecht & Stone, 2015), and humanitarian assistance and disaster response (Meier, 2015), where one agent should collaborate and/or compete with all other agents. When directly applying the existing MARL algorithms to complex games with a large number of agents, as we will show in Sec. 5.3, the agents may fail to learn good strategies and end up with little interaction with other agents even when collaboration is significantly beneficial. Yang et al. (2018) proposed a provably-converged meanfield formulation to scale up the actor-critic framework by feeding the state information and the average value of nearby agents’ actions to the critic. However, this formulation strongly relies on the assumption that the value function for each agent can be well approximated by the mean of local
∗Equal contribution †Equal advising
pairwise interactions. This assumption often does not hold when the interactions between agents become complex, leading to a significant drop in the performance.
In this paper, we propose a general learning paradigm called Evolutionary Population Curriculum (EPC), which allows us to scale up the number of agents exponentially. The core idea of EPC is to progressively increase the population of agents throughout the training process. Particularly, we divide the learning procedure into multiple stages with increasing number of agents in the environment. The agents first learn to play in simpler scenarios with less agents and then leverage these experiences to gradually adapt to later stages with more agents and ultimately our desired population.
There are two key components in our curriculum learning paradigm. To process the varying number of agents during the curriculum procedure, the policy/critic needs to be population-invariant. So, we choose a self-attention (Vaswani et al., 2017) based architecture which can generalize to an arbitrary number of agents with a fixed number of parameters. More importantly, we introduce an evolutionary selection process, which helps address the misalignment of learning goals across stages and improves the agents’ performance in the target environment. Intuitively, our within-stage MARL training objective only incentivizes agents to overfit a particular population in the current stage. When moving towards a new stage with a larger population, the successfully trained agents may not adapt well to the scaled environment. To mitigate this issue, we maintain multiple sets of agents in each stage, evolve them through cross-set mix-and-match and parallel MARL fine-tuning in the scaled environment, and select those with better adaptability to the next stage.
EPC is RL-algorithm agnostic and can be potentially integrated with most existing MARL algorithms. In this paper, we illustrate the empirical benefits of EPC by implementing it on a popular MARL algorithm, MADDPG (Lowe et al., 2017), and experimenting on three challenging environments, including a predator-prey-style individual survival game, a mixed cooperative-and-competitive battle game, and a fully cooperative food collection game. We show that EPC outperforms baseline approaches by a large margin on all these environments as the number of agents grows even exponentially. We also demonstrate that our method can improve the stability of the training procedure.

heading: 2 RELATED WORK

text:
Multi-Agent Reinforcement Learning: It has been a long history in applying RL to multi-agent games (Littman, 1994; Shoham et al., 2003; Panait & Luke, 2005; Wright et al., 2019). Recently, deep RL techniques have been applied into the multi-agent scenarios to solve complex Markov games and great algorithmic advances have been achieved. Foerster et al. (2016) and He et al. (2016) explored a multi-agent variant of deep Q-learning; Peng et al. (2017) studied a fully centralized actor-critic variant; Foerster et al. (2018) developed a decentralized multi-agent policy gradient algorithm with a centralized baseline; Lowe et al. (2017) proposes the MADDPG algorithm which extended DDPG to the multi-agent setting with decentralized policies and centralized Q functions. Our population curriculum approach is a general framework for scaling MARL which can be potentially combined with any of these algorithms. Particularly, we implement our method on top of the MADDPG algorithm in this paper and take different MADDPG variants as baselines in experiments. There are also other works studying large-scale MARL recently (Lin et al., 2018; Jiang & Lu, 2018; Yang et al., 2018; Suarez et al., 2019), which typically simplify the problem by weight sharing and taking only local observations. We consider a much more general setting with global observations and unsharedweight agents. Additionally, our approach is a general learning paradigm which is complementary to the specific techniques proposed in these works.
Attention-Based Policy Architecture: Attention mechanism is widely used in RL policy representation to capture object level information (Duan et al., 2017; Wang et al., 2018), represent relations (Zambaldi et al., 2018; Malysheva et al., 2018; Yang et al., 2019) and extract communication channels (Jiang & Lu, 2018). Iqbal & Sha (2019) use an attention-based critic. In our work, we utilize an attention module in both policy and critic, inspired by the transformer architecture (Vaswani et al., 2017), for the purpose of generalization to an arbitrary number of input entities.
Curriculum Learning: Curriculum learning can be tracked back to Elman (1993), and its core idea is to “start small”: learn the easier aspects of the task first and then gradually increase the task difficulty. It has been extended to deep neural networks on both vision and language tasks (Bengio et al., 2009) and much beyond: Karras et al. (2017) propose to progressively increase the network capacity for synthesizing high quality images; Murali et al. (2018) apply a curriculum over the control space for robotic manipulation tasks; several works (Wu & Tian, 2016; Florensa et al., 2017; Sukhbaatar et al., 2017; Wang et al., 2019) have proposed to first train RL agents on easier goals
and switch to harder ones later. Baker et al. (2019) show that multi-agent self-play can also lead to autocurricula in open-ended environments. In our paper, we propose to progressively increase the number of the agents as a curriculum for better scaling multi-agent reinforcement learning.
Evolutionary Learning: Evolutionary algorithms, originally inspired by Darwin’s natural selection, has a long history (Bäck & Schwefel, 1993), which trains a population of agents in parallel, and let them evolve via crossover, mutation and selection processes. Recently, evolutionary algorithms have been applied to learn deep RL policies with various aims, such as to enhance training scalability (Salimans et al., 2017), to tune hyper-parameters (Jaderberg et al., 2017), to evolve intrinsic dense rewards (Jaderberg et al., 2018), to learn a neural loss for better generalization (Houthooft et al., 2018), to obtain diverse samples for faster off-policy learning (Khadka & Tumer, 2018), and to encourage exploration (Conti et al., 2018). Leveraging this insight, we apply evolutionary learning to better scale MARL: we train several groups of agents in each curriculum stage and keep evolving them to larger populations for the purpose of better adaptation towards the desired population scale and improved training stability. Czarnecki et al. (2018) proposed a similar evolutionary mix-and-match training paradigm to progressively increase agent capacity, i.e., larger action spaces and more parameters. Their work considers a fixed environment with an increasingly more complex agent and utilizes the traditional parameter crossover and mutation during evolution. By contrast, we focus on scaling MARL, namely an increasingly more complex environment with a growing number of agents. More importantly, we utilize MARL fine-tuning as an implicit mutation operator rather than the classical way of mutating parameters, which is more efficient, guided and applicable to even a very small number of evolution individuals. A similar idea of using learning for mutation is also considered by Gangwani & Peng (2018) in the single-agent setting.

heading: 3 BACKGROUND

text:
Markov Games: We consider a multi-agent Markov decision processes (MDPs) (Littman, 1994). Such an N -agent Markov game is defined by state space S of the game, action spacesA1, ...,AN and observation spaces O1, ...,ON for each agent. Each agent i receives a private observation correlated with the state oi : S 7→ Oi and produces an action by a stochastic policy πθi : Oi × Ai 7→ [0, 1] parameterized by θi. Then the next states are produced according to the transition function T : S ×A1 × ...×AN 7→ S . The initial state is determined by a distribution ρ : S 7→ [0, 1]. Each agent i obtains rewards as a function of the state and its action ri : S ×Ai 7→ R, and aims to maximize its own expected return Ri = ∑T t=0 γ trti(s t, ati), where γ is a discount factor and T is the time horizon. To minimize notation, we omit subscript of policy when there is no ambiguity.
Multi-Agent Deep Deterministic Policy Gradient (MADDPG): MADDPG (Lowe et al., 2017) is a multi-agent variant of the deterministic policy gradient algorithm (Silver et al., 2014). It learns a centralized Q function for each agent which conditions on global state information to resolve the non-stationary issue. Consider N agents with deterministic policies µ = {µ1, ...,µN} where µi : Oi 7→ Ai is parameterized by θi. The policy gradient for agent i is:
∇θiJ(θi) = Ex,a∼D[∇θiµi(oi)∇aiQ µ i (x, a1, ..., aN )|ai=µi(oi)], (1)
Here D denotes the replay buffer while Qµi (x, a1, ..., aN ) is a centralized action-value function for agent i that takes the actions of all agents, a1, . . . , aN and the state information x (i.e., x = (o1, ..., oN ) or simply x = s if s is available). Let x′ denote the next state from the environment transition. The replay bufferD contains experiences in the form of tuples (x,x′, a1, . . . , aN , r1, . . . , rN ). Suppose the centralized critic Qµi is parameterized by φi. Then it is updated via:
L(φi) = Ex,a,r,x′ [(Qµi (x, a1, . . . , aN )− y) 2], y = ri + γ Q
µ′
i (x ′, a′1, . . . , a ′ N ) ∣∣ a′j=µ ′ j(oj) , (2)
where µ′ = {µθ′1 , ...,µθ′N } is the set of target policies with delayed parameters θ ′ i. Note that the centralized critic is only used during training. At execution time, each policy µθi remains decentralized and only takes local observation oi.

heading: 4 EVOLUTIONARY POPULATION CURRICULUM

text: In this section, we will first describe the base network architecture with the self-attention mechanism (Vaswani et al., 2017) which allows us to incorporate a flexible number of agents during training. Then we will introduce the population curriculum paradigm and the evolutionary selection process.

heading: 4.1 POPULATION-INVARIANT ARCHITECTURE

text:
We describe our choice of architecture based on the MADDPG algorithm (Lowe et al., 2017), which is population-invariant in the sense that both the Q function and the policy can take in an arbitrary number of input entities. We first introduce the Q function (Fig. 1) and then the policy.
We adopt the decentralized execution framework, so each agent has its own Q function and policy network. Particularly for agent i, its centralized Q function is represented as follows:
Qµi (x, a1, . . . , aN ) = hi([gi(fi(oi, ai)), vi]), where vi = attention(fi(oj , aj)∀j 6= i) (3)
Here fi(oj , aj) is an observation-action encoder (the green box in Fig. 1(a)) which takes in the observation oj and the action aj from agent j, and outputs the agent embedding of agent j; vi denotes the global attention embedding (the orange box in Fig. 1(a)) over all the agent embeddings. We will explain vi and fi later. gi is a 1-layer fully connected network processing the embedding of the ith agent’s own observation and action. hi is a 2-layer fully connected network that takes the concatenation of the output of gi and the global attention embedding vi and outputs the final Q value.
Attention Embedding vi: We define the attention embedding vi by a weighted sum of each agent’s embedding fi(oj , aj) for j 6= i:
vi = ∑ j 6=i αi,jfi(oj , aj) (4)
The coefficient αi,j is computed by
αi,j = exp (βi,j)∑ j 6=i exp (βi,j) , βi,j = f T i (oi, ai)W T ψWφfi(oj , aj) (5)
where Wψ and Wφ are parameters to learn. βi,j computes the correlation between the embeddings of agent i and every other agent j via an inner product. αi,j is then obtained by normalizing βi,j by a softmax function. Since we represent the observations and actions of other agents with a weighted mean vi from Eq. 4, we can model the interactions between agent i and an arbitrary number of other agents, which allows us to easily increase the number of agents in our curriculum training paradigm.
Observation-Action Encoder fi: We now define the structure of fi(oj , aj) (Fig. 1(b)). Note that the observation of agent j, oj , also includes many entities, i.e., states of all visible agents and objects in the game. Suppose oj contains M entities, i.e., oj = [oj,1, . . . , oj,M ]. M may also vary as the agent population scales over training procedure or simply during an episode when some agents die. Thus, we apply another attention module to combine these entity observations together in a similar way to how vi is computed (Eq. 4, 5).
In more details, we first apply an entity encoder for each entity type to obtain entity embeddings of all the entities within that type. For example, in oj , we can have embeddings for agent entities (green boxes in Fig. 1(b)) and landmark/object entities (purple boxes in Fig. 1(b)). Then we apply an attention module over each entity type by attending the entity embedding of agent j to all the entities of this type to obtain an attended type embedding (the orange box in Fig. 1(b)). Next, we concatenate
all the type embeddings together with the entity embedding of agent j as well as its action embedding. Finally, this concatenated vector is forwarded to a fully connected layer to generate the output of fi(oj , aj). Note that in the overall critic network of agent i, the same encoder fi is applied to every observation-action pair so that the network can maintain a fixed size of parameters even when the number of agents increases significantly.
Policy Network: The policy network µi(oi) has a similar structure as the observation-action encoder fi(oi, ai), which uses an attention module over the entities of each type in the observation oi to adapt to the changing population during training. The only difference in this network is that the action ai is not included in the input. Notably, we do not share parameters between the Q function and the policy.

heading: 4.2 POPULATION CURRICULUM

text:
We propose to progressively scale the number of agents in MARL with a curriculum. Before combining with the evolutionary selection process, we first introduce a simpler version, the vanilla population curriculum (PC), where we perform the following stage-wise procedure: (i) the initial stage starts with MARL training over a small number of agents using MADDPG and our populationinvariant architecture; (ii) we start a new stage and double1 the number of agents by cloning each of the existing agents; (iii) apply MADDPG training on this scaled population until convergence; (iv) if the desired number of agents is not reached, go back to step (ii).
Mathematically, given N trained agents with parameters θ = {θ1, ..., θN} from the previous stage, we want to increase the number of the agents to 2N with new parameters θ̃ = {θ̃1, ..., θ̃N , ..., θ̃2N} for the next stage . In this vanilla version of population curriculum, we simply initialize θ̃ by setting θ̃i ← θi and θ̃N+i ← θi, and then continue MADDPG training on θ̃ to get the final policies for the new stage. Although θ̃i and θ̃N+i are both initialized from θi, as training proceeds, they will converge to different policies since these policies are trained in a decentralized manner in MADDPG.

heading: 4.3 EVOLUTIONARY SELECTION

text:
Introducing new agents by directly cloning existing ones from the previous stage has a clear limitation: the policy parameters suitable for the previous environment are not necessarily the best initialization for the current stage as the population is scaled up. In the purpose of better performance in the final game with our desired population, we need to promote agents with better adaptation abilities during early stages of training.
Therefore, we propose an evolutionary selection process to facilitate the agents’ scaling adaption ability during the curriculum procedure. Instead of training a single set of agents, we maintain K parallel sets of agents in each stage, and perform crossover, mutation and selection among them for the next stage. This is the last piece in our proposed Evolutionary Population Curriculum (EPC) paradigm, which is essentially population curriculum enhanced by the evolutionary selection process.
Specifically, we assume the agents in the multi-agent game have Ω different roles. Agents in the same role have the same action set and reward structure. For example, we have Ω = 2 roles in a predator-prey game, namely predators and prey, and Ω = 1 role of agents for a fully cooperative game with homogeneous agents. For notation conciseness, we assume there are N1 agents of role 1, namely A1 = {µ1, ...,µN1}; N2 agents of role 2, namely A2 = {µN1+1, ...,µN1+N2}, and so on. In each stage, we keep K parallel sets for each role of agents, denoted by A(1)i , . . . , A (K) i for role i, and take a 3-step procedure, i.e., mix-and-match (crossover), MARL fine-tuning (mutation) and selection, as follows to evolve these K parallel sets of agents for the next stage.
Mix-and-Match (Crossover): In the beginning of a curriculum stage, we scale the population of agents from N to 2N . Note that we have K parallel agent sets of size Ni for role i, namely A
(1) i , . . . , A (K) i . We first perform a mix-and-match over these parallel sets within every role i: for each set A(j)i , we pair it with all the K sets of the same role, which leads to K(K + 1)/2 new scaled agent sets of size 2Ni. Given these scaled sets of agents, we then perform another mix-and-match across all the Ω roles: we pick one scaled set for each role and combine these Ω selected sets to produce a scaled game with 2N agents. For example, in the case of Ω = 2, we can pick one agent set A(k1)1 from the first role and another agent set A (k2) 2 from the second role to form a scaled game.
1Generally, we can scale up the population with any constant factor by introducing any amount of cloned agents. We use the factor of 2 as a concrete example here for easier understanding.
Thus, there are Cmax = (K(K + 1)/2) Ω different combinations in total through this mix-and-match process. We sample C games from these combinations for mutation in the next step. Since we are mixing parallel sets of agents, this process can be considered as the crossover operator in standard evolutionary algorithms.
MARL Fine-Tuning (Mutation): In standard evolutionary algorithms, mutations are directly performed on the parameters, which is inefficient in high-dimensional spaces and typically requires a large amount of mutants to achieve sufficient diversity for evolution. Instead, here we adopt MARL fine-tuning in each curriculum stage (step (iii) in vanilla PC) as our guided mutation operator, which naturally and efficiently explores effective directions in the parameter space. Meanwhile, due to the training variance, MARL also introduces randomness which benefits the overall diversity of the evolutionary process. Concretely, we apply parallel MADDPG training on each of the C scaled games generated from the mix-and-match step and obtain C mutated sets of agents for each role.
Selection: Among these C mutated sets of agents for each role, only the best K mutants can survive. In the case of Ω = 1, the fitness score of a set of agents is computed as their average reward after MARL training. In other cases when Ω ≥ 2, given a particular mutated set of agents of a specific role, we randomly generate games for this set of agents and other mutated sets from different agent roles. We take its average reward from these randomly generated games as the fitness score for this mutated set. We pick the top-K scored sets of agents in each role to advance to the next curriculum stage.
Algorithm 1: Evolutionary Population Curriculum Data: environment E(N, {Ai}1≤i≤Ω) with N agents of Ω roles, desired population Nd, initial population N0, evolution size K, mix-and-match size C Result: a set of Nd best policies N ← N0; initialize K parallel agent sets A(1)i , . . . , A (K) i for each role 1 ≤ i ≤ Ω; initial parallel MARL training on K games, E(N, {A(j)i }1≤i≤Ω) for 1 ≤ j ≤ K; while N < Nd do
N ← 2×N ; for 1 ≤ j ≤ C do
for each role 1 ≤ i ≤ Ω: j1, j2 ← unif(1,K), Ã(j)i ← A (j1) i +A (j2) i (mix-and-match);
MARL training in parallel on E(N, {Ã(j)i }1≤i≤Ω) for 1 ≤ j ≤ C (guided mutation) ; for role 1 ≤ i ≤ Ω do
for 1 ≤ j ≤ C do S
(j) i ← Ekt 6=i∼[1,C] [ avg. rewards on E(N, {Ã(k1)1 , . . . , Ã (j) i , . . . , Ã (kΩ) Ω }) ] (fitness);
A (1) i , . . . , A (K) i ← top-K w.r.t. Si from Ã (1) i , . . . , Ã (C) i (selection);
return the best set of agents in each role, i.e., {A(k ? i )
i |k?i ∈ [1,K] ∀1 ≤ i ≤ Ω};
Overall Algorithm: Finally, when the desired population is achieved, we take the best set of agents in each role based on their last fitness scores as the output. We conclude the detailed steps of EPC in Alg. 1. Note that in the first curriculum stage, we just train K parallel games without mix-and-match or mutation. So, EPC simply selects the best from the K initial sets in the first stage while the evolutionary selection process only takes effect starting from the second stage. We emphasize that although we evolve multiple sets of agents in each stage, the three operators, mix-and-match, MARL fine-tuning and selection, are all perfectly parallel. Thus, the evolutionary selection process only introduces little influence on the overall training time. Lastly, EPC is an RL-algorithm-agnostic learning paradigm that can be potentially integrated with any MARL algorithm other than MADDPG.

heading: 5 EXPERIMENT

text: We experiment on three challenging environments, including a predatory-prey-style Grassland game, a mixed-cooperative-and-competitive Adversarial Battle game and a fully cooperative Food Collection game. We compare EPC with multiple baseline methods on these environments with different scales of agent populations and show consistently large gains over the baselines. In the following, we will first introduce the environments and the baselines, and then both qualitative and quantitative performances of different methods on all three environments.

heading: 5.1 ENVIRONMENTS

text:
All these environments are built on top of the particle-world environment (Mordatch & Abbeel, 2018) where agents take actions in discrete timesteps in a continous 2D world.
Grassland: In this game, we have Ω = 2 roles of agents, NS sheep and NW wolves, where sheep moves twice as fast as wolves. We also have a fixed amount of L grass pellets (food for sheep) as green landmarks (Fig. 2a). A wolf will be rewarded when it collides with (eats) a sheep,
and the (eaten) sheep will obtain a negative reward and becomes inactive (dead). A sheep will be rewarded when it comes across a grass pellet and the grass will be collected and respawned in another random position. Note that in this survival game, each individual agent has its own reward and does not share rewards with others. Adversarial Battle: This scenario consists of L units of resources as green landmarks and two teams of agents (i.e., Ω = 2 for each team) competing for the resources (Fig. 2b). Both teams have the same number of agents (N1 = N2). When an agent collects a unit of resource, the resource will be respawned and all the agents in its team will receive a positive reward. Furthermore, if there are more than two agents from team 1 collide with one agent from team 2, the whole team 1 will be rewarded while the trapped agent from team 2 will be deactivated (dead) and the whole team 2 will be penalized, and vice versa. Food Collection: This game has N food locations and N fully cooperative agents (Ω = 1). The agents need to collaboratively occupy as many food locations as possible within the game horizon (Fig. 2c). Whenever a food is occupied by any agent, the whole team will get a reward of 6/N in that timestep for that food. The more food occupied, the more rewards the team will collect.
In addition, we introduce collision penalties as well as auxiliary shaped rewards for each agent in each game for easier training. All the environments are fully observable so that each agent needs to handle a lot of entities and react w.r.t. the global state. More environment details are in Appx. A.

heading: 5.2 METHODS AND METRIC

text:
We evaluate the following approaches in our experiments: (1) the MADDPG algorithm (Lowe et al., 2017) with its original architecture (MADDPG); (2) the provably-converged mean-field algorithm (Yang et al., 2018) (mean-field); (3) the MADDPG algorithm with our population-invariant architecture (Att-MADDPG); (4) the vanilla population curriculum without evolutionary selection (vanilla-PC); and (5) our proposed EPC approach (EPC). For EPC parameters, we choose K = 2 for Grassland and Adversarial Battle and K = 3 for Food Collection; for the mix-and-match size C, we simply set it Cmax and enumerate all possible mix-and-match combinations instead of random sampling. All the baseline methods are trained until the same amount of accumulative episodes as EPC took. More training details can be found in Appx. B.
For Grassland and Adversarial Battle with Ω = 2, we evaluate the performance of different methods by competing their trained agents against our EPC trained agents. Specifically, in Grassland, we let sheep trained by each approach compete with the wolves from EPC and collect the average sheep reward as the evaluation metric for sheep. Similarly, we take the same measurement for wolves from each method. In Adversarial Battle, since two teams are symmetric, we just evaluate the shared reward of one team trained by each baseline against another team by EPC as the metric. For Food Collection with Ω = 1, since it is fully cooperative, we take the team reward for each method as the evaluation metric. In addition, for better visualization, we plot the normalized scores by normalizing the rewards of different methods between 0 and 1 in each scale for each game. More evaluation details are in Appx. C.

heading: 5.3 QUALITATIVE RESULTS

text:
In Grassland, as the number of wolves goes up, it becomes increasingly more challenging for sheep to survive; meanwhile, as the sheep become more intelligent, the wolves will be incentivized to be more aggressive accordingly. In Fig. 3, we illustrate two representative matches for competition, including one using the MADDPG sheep against the EPC wolves (Fig. 3a), and the other between the EPC sheep and the MADDPG wolves (Fig. 3b). From Fig. 3a, we can observe that the MADDPG sheep can be easily eaten up by the EPC wolves (note that dark circle means the sheep is eaten). On
(a) EPC (b) MADDPG
Figure 4: Adversarial Battle: dark particles are dead agents. (a) EPC (b) MADDPG
Figure 5: Food Collection
the other hand, in Fig. 3b, we can see that the EPC sheep learns to eat the grass and avoid the wolves at the same time.
In Adversarial Battle, we visualize two matches in Fig. 4 with one over agents by EPC (Fig. 4a) and the other over agents by MADDPG (Fig. 4b). We can clearly see the collaborations between the EPC agents: although the agents are initially spread over the environment, they learn to quickly gather as a group to protect themselves from being killed. While for the MADDPG agents, their behavior shows little incentives to cooperate or compete — these agents stay in their local regions throughout the episode and only collect resources or kill enemies very infrequently.
In Food Collection (Fig. 5), the EPC agents in Fig. 5a learn to spread out and occupy as many food as possible to maximize the team rewards. While only one agent among the MADDPG agents in Fig. 5b successfully occupies a food in the episode.

heading: 5.4 QUANTITATIVE RESULTS

text:
Quantitative Results in Grassland In the Grassland game, we perform curriculum training by starting with 3 sheep and 2 wolves, and gradually increase the population of agents. We denote a game with NS sheep and NW wolves by “scale NS-NW ”. We start with scale 3-2 and gradually increase the game size to scales 6-4, 12-8 and finally 24-16.For the two curriculum learning approach, vanilla-PC and EPC, we train over 105 episodes in the first curriculum stage (scale 3-2) and fine-tune the agents with 5× 104 episodes after mix-and-match in each of the following stage. For other methods that train the agents from scratch, we take the same accumulative training iterations as the curriculum methods for a fair comparison.
Main Results: We report the performance of different methods for each game scale in Fig. 6a. Overall, there are little differences between the mean-field approach and the original MADDPG algorithm while the using the population-invariant architecture (i.e., Att-MADDPG) generally boosts the performance of MADDPG. For the method with population curriculum, vanilla-PC performs almost the same as training from scratch (Att-MADDPG) when the number of agents in the environment is small (i.e., 6-4) but the performance gap becomes much more significant when the population further grows (i.e., 12-18 and 24-16). For our proposed EPC method, it consistently outperforms all the baselines across all the scales. Particularly, in the largest scale 24-16, EPC sheep receive 10x more rewards than the best baseline sheep without curriculum training.
Detailed Statistics: Besides rewards, we also compute the statistics of sheep to understand how the trained sheep behave in the game. We perform competitions between sheep trained by different methods against the EPC wolves and measure the average number of total grass pellets eaten per episode, i.e, #grass eaten, and the average percentage of sheep that survive until the end of an episode, i.e., survival rate, in Fig. 6b. We can observe that as the population increases, it becomes increasingly harder for sheep to survive while EPC trained sheep remain a high survival rate even on the largest scale. Moreover, as more sheep in the game, EPC trained sheep consistently learn to eat more grass even under the strong pressure from wolves. In contrast, the amount of eaten grass of MADDPG approach (i.e., Att-MADDPG) drastically decreases when the number of wolves becomes large.
Quantitative Results in Adversarial Battle
In this game, we evaluate on environments with different sizes of agent population N , denoted by scale N1-N2 where N1 = N2 = N/2. We start the curriculum from scale 4-4 and the increase the population size to scale 8-8 (N = 16) and finally 16-16 (N = 32). Both vanilla-PC and EPC take 5× 104 training episodes in the first stage and then 2 × 104 episodes in the following two curriculum stages. We report the normalized scores of different methods in Fig. 7, where agents trained by EPC outperforms all the baseline methods increasingly more significant as the agent population grows. Quantitative Results in Food Collection
In this game, we begin curriculum training with N = 3, namely 3 agents and 3 food locations, and progressively increase the population size N to 6, 12 and finally 24. Both vanilla-PC and EPC perform training on 5 × 104 episodes on the first stage of N = 3 and then 2× 104 episodes in each of the following curriculum stage. We report the normalized scores for all the methods in Fig. 8, where EPC is always the best among all the approaches with a clear margin. Note that the performance of the original MADDPG and the meanfield approach drops drastically as the population size N increases. Particularly, the mean-field approach performs even worse than the
original MADDPG method. We believe this is because in this game, the agents must act according to the global team state collaboratively, which means the local approximation assumption in the mean-field approach does not hold clearly.
Ablative Analysis Stability Analysis: The evolutionary selection process in EPC not only leads to better final performances but also stabilizes the training procedure. We validate the stability of EPC by computing the variance over 3 training seeds for the same experiment and comparing with the variance of vanilla-PC, which is also obtained from 3 training seeds. Specifically, we pick the second stage of curriculum learning and visualize the variance of agent scores throughout the stage of training. These scores are computed by competing against the final policy trained by EPC. We perform analysis on all the 3 environments: Grassland with scale 6-4 (Fig. 9a), Adversarial Battle with scale 8-8 (Fig. 9b) and Food Collection with scale 6 (Fig. 9c). We can observe that the variance of EPC is much smaller than vanilla-PC in different games. Convergence Analysis: To illustrate that the self-attention based policies trained from a smaller scale is able to well adapt to a larger scale via fine-tuning, we pick a particular mutant by EPC in the second curriculum stage and visualize its learning curve throughout fine-tuning for all the environments, Grassland (Fig. 9d), Adversarial Battle (Fig. 9e) and Food Collection (Fig. 9f). The scores are computed in the same way as the stability analysis. By comparing to MADDPG and Att-MADDPG, which train policies from scratch, we can see that EPC starts learning with a much higher score, continues to improve during fine-tuning and quickly converges to a better solution. Note that all baselines are in fact trained much longer. The full convergence curves are in App. D.1. Generalization: We investigate whether the learned policies can generalize to a different test environment with even a larger scale than the training ones. To do so, we take the best polices trained by different methods on the largest population and directly apply these policies to a new environment with a doubled population by self-cloning. We evaluate in all the environments with EPC, vanilla-PC
and Att-MADDPG and measure the normalized scores of different methods, which is computed in the same way as the fitness score. In all cases, we observe a large advantage of EPC over the other two methods, indicating the better generalization ability for policies trained by EPC.

heading: 6 CONCLUSION

text: In this paper, we propose to scale multi-agent reinforcement learning by using curriculum learning over the agent population with evolutionary selection. Our approach has shown significant improvements over baselines not only in the performance but also the training stability. Given these encouraging results on different environments, we believe our method is general and can potentially benefit scaling other MARL algorithms. We also hope that learning with a large population of agents can also lead to the emergence of swarm intelligence in environments with simple rules in the future.

heading: ACKNOWLEDGMENT

text: This research is supported in part by ONR MURI N000141612007, ONR Young Investigator to AG. FF is also supported in part by NSF grant IIS-1850477, a research grant from Lockheed Martin, and the U.S. Army Combat Capabilities Development Command Army Research Laboratory Cooperative Agreement Number W911NF-13-2-0045 (ARL Cyber Security CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the funding agencies. We also sincerely thank Bowen Baker and Ingmar Kanitscheider from OpenAI for valuable suggestions and comments.

heading: A ENVIRONMENT DETAILS

text:
In the Grassland game, sheep gets +2 reward when he eats the grass, -5 reward when eaten by wolf. The wolf get +5 reward when eats a sheep. We also shape the reward by distance, sheep will get less negative reward when it is closer to grass and wolf will get less negative reward when it is closer to sheep. This game is adapted from the original Predator-prey game in the MADDPG paper (Lowe et al., 2017) by introducing grass and allowing agent to die.
In the Adversarial Battle game, agent will get +1 reward when he eats the food, −6 reward when killed by other agents. If N agents kill an enemy, they will be rewarded +6/N . We shape the reward by distance. Agent will receive less negative rewards when it is closer to other agents and grass. We want to encourage collision within agents and also will be easier for them to learn to eat. This game is adapted from the mean-field MARL paper (Yang et al., 2018) by converting it from a grid world to particle-world, introducing food and only allowing 2-agent cooperative killing.
In the Food Collection game, there are N agents and N food locations. Each agent will get a shared +6/N reward per timestep when one food is occupied by any agent. If one agent gets collision with another, all of the agents will get a punish of −6/N . We shape the reward by distance. Agents will receive less negative rewards when it gets closer to the food. Since the number of agents and food are equal, we want to avoid the collision within agents and let the agents to learn to occupy as many food as possible. This is exactly the same game as the Cooperative Navigation game in the MADDPG paper. We slightly change the reward function to ensure it is bounded w.r.t. arbitrary number of agents.
We use the normalized reward as the score during evaluation. For a particular game with a particular scale, we first collect the reward for each type of agents, namely the average reward of each individual of that type without the shaped rewards. Then we re-scale the collected rewards by considering the lowest reward among all methods as score 0 and highest reward as score 1.

heading: B TRAINING DETAILS

text:
We follow all the hyper-parameters in the original MADDPG paper (Lowe et al., 2017) for both EPC and all the baseline methods considered. Particularly, we use the Adam optimizer with learning rate 0.01, β1 = 0.9, β2 = 0.999 and ε = 10−8 across all experiments. τ = 0.01 is set for target network update and γ = 0.95 is used as discount factor. We also use a replay buffer of size 106 and we update the network parameters after every 100 samples. The batch size is 1024. All the baseline methods are trained for a number of episodes that equals the accumulative number of episodes that EPC has taken.
We set K = 2 in all the games during training except that K = 3 in the food collection game due to computational constraints. During EPC training, in the grassland game, we train the scale of 3 sheep 2 wolf for 100000 episodes. We train another 50000 episodes every time the agents number doubles. In the adversarial battle game and food collection game, we train the first scale for 50000 episodes. We train another 20000 episodes every time the agents number doubles.
In the grassland game, the entity types are the agent itself, other sheep, other wolf and food. We thus have four types of entity encoders for each of those entity types. In the adversarial battle game, Similar to grassland game, the entity types are agent itself, other teammates, enemies and food. We also have four types of entity encoders for each of those entity types. Since there is only one group in the food collection game, the entity types are agent itself, other teammates and food. We thus have three entity encoders in our network.

heading: C EVALUATION DETAILS

text:
To evaluate the agents trained in the environment with Ω = 2, we make two roles of agents trained with different approaches compete against each other. Each competition is simulated for 10000 episodes. The average normalized reward over the 10000 episodes will be used as the competition score for each side. Note that in our experiments, we let all the methods compete against our EPC approach for evaluation. For adversarial battle game, we take the average score of two teams as the model’s final evaluation score, since the two teams in this game are completely symmetric.
In the food collection game, since there is only one role, we simply simulate the model for 10000 episodes. The average normalized reward over the 10000 episodes will be used as the score of the model.

heading: D ADDITIONAL DETAILS ON EXPERIMENT RESULTS

text:
D.1 FULL TRAINING CURVES FOR BASELINES All the baseline methods are trained for a number of episodes that equals the accumulative number of episodes that EPC has taken.
The purpose of Figure 9e,9e,9f is simply showing the transfer performance, i.e., the initialization produced by EPC from the previous stage is effective and can indeed leverage past experiences to warm-start. The x-axis of
the plot was shrunk for visualization purpose. Here we illustrate the complete convergence curve of baselines, i.e., ATT-MADDPG and MADDPG, in Figure 11a,11b,11c for the 3 games respective. Although Att-MADDPG takes a much longer time to learn, its performance is still far worse than EPC.
D.2 RAW REWARD NUMBERS OF EVALUATION RESULTS In this section, we provide the actual rewards without normalization when comparing all the baselines with EPC. These scores are corresponding to the histograms reported in Figure 6a, 7, 8.
Grassland game, wolf rewards, corresponding to wolf in Figure 6a:
scale MADDPG mean field Att-MADDPG vanilla-PC EPC 3-2 0.596 0.877 0.8145 0.8145 1.407 6-4 3.7735 0.9515 2.5905 2.001 3.7735
12-8 3.1915 3.385 10.2125 9.974 14.377 24-16 14.482 18.272 32.8945 47.6365 61.4245
Grassland game, sheep rewards, corresponding to sheep in Figure 6a:
scale MADDPG mean field Att-MADDPG vanilla-PC EPC 3-2 -4.0026 -3.9947 2.66 2.66 8.3846 6-4 -20.2494 -20.5107 0.9892 1.1804 10.1455 12-8 -52.863 -53.6338 -42.4801 -11.3736 3.3774
24-16 -119.1327 -118.5668 -111.0656 -70.1981 -44.1031
Adversarial Battle game, rewards of team 1, corresponding to Figure 7:
scale MADDPG mean field Att-MADDPG vanilla-PC EPC 4-4 7.51555 5.81165 22.6357 22.6357 26.86355 8-8 0.6692 -0.58115 43.7801 46.89595 65.75585
16-16 -46.6398 -35.5978 28.8336 109.4406 189.69775
Food Collection game, team rewards, corresponding to Figure 8:
scale MADDPG mean field Att-MADDPG vanilla-PC EPC 3 55.06 42.74 61.6488 61.6488 64.822 6 17.01 3.37 49.3626 58.0014 63.7004
12 6.32 6.735 49.45755 52.3625 59.54 24 10.346075 7.830975 33.435 49.998025 59.47035
D.3 PAIRWISE COMPETITION RESULTS BETWEEN ALL METHODS IN COMPETITIVE GAMES For visualization purpose, we only illustrate the scores of the competitions between baselines and EPC in the main paper. Here we provide the complete competition rewards between every pair of methods in both Grassland and Adversarial Battle with the largest population of agents as follows.
Here show the wolf rewards in Grassland with scale 24-16. For wolves trained by each approach, we compare them against the sheep by all the methods. EPC wolves always have the highest rewards as in the bottom row. Correspondingly, when different wolves compete against EPC sheep, they always obtain the lowest rewards as in the rightmost column.
wolf sheep MADDPG mean-field Att-MADDPG Vanilla-PC EPC
MADDPG 66.914 67.0945 66.34 23.048 14.482 mean-field 75.7655 74.23 74.7375 28.3705 18.272
Att-MADDPG 103.22 103.326 98.07 49.557 32.8945 Vanilla-PC 110.333 111.3735 101.8975 64.53 47.6365
EPC 120.9025 121.4325 115.956 82.381 61.4245
Here show the sheep rewards in Grassland with scale 24-16. For sheep trained by each approach, we compete them against the all different wolves. EPC sheep always have the highest rewards as in the last row. Correspondingly, when competing different sheep against EPC wolves, the rewards are always the lowest as in the rightmost column.
sheep wolf MADDPG mean-field Att-MADDPG Vanilla-PC EPC
MADDPG -63.5096 -72.5443 -100.4636 -107.825 -119.1327 mean-field -63.7089 -71.0714 -100.6304 -108.8917 -118.5668 Att-MADDPG -62.9416 -71.3339 -95.0522 -99.1207 -111.0656 Vanilla-PC -5.7086 -7.8011 -31.9936 -49.5186 -70.1981
EPC 9.2135 10.5892 -6.9846 -27.3475 -44.1031
Here we show the rewards of team 1 in Adversarial Battle with scale 16-16. For agents trained by each approach, we compare them as team 1 against all different methods as team 2. When EPC agents as team 1, no matter which opponent is, they always get the highest rewards as in the last row. When other methods compete against EPC, the obtained rewards are always the lowest as in the rightmost column.
reported compared MADDPG mean-field Att-MADDPG Vanilla-PC EPC
MADDPG 61.4555 17.1591 2.8033 -29.9242 -46.6398 mean-field 104.41315 59.01315 27.9004 11.1891 -35.5978 Att-MADDPG 146.9829 117.3804 81.702 18.57425 28.8336 Vanilla-PC 202.9163 155.2805 174.91965 123.7212 109.4406
EPC 339.63255 318.3223 256.8464 198.3621 189.69775
D.4 VARIANCE OF PERFORMANCE EVALUATIONS We present the performance of all approaches in all three games with the largest scale. We train all the approaches with 3 different seeds and show the normalized scores with variance as following. We can see that EPC not only gives better results but also much smaller variance.

heading: E ADDITIONAL EXPERIMENTS ON THE ORIGINAL PREDATOR-PREY GAME

text:
Grassland is adapted from the Predator-prey game introduced by the original MADDPG paper (Lowe et al., 2017). To further validate our empirical results, we additionally study the performances of different algorithms on the unmodified Predator-prey game as follows.
(a) Normalized scores with variances in Grassland in scale 24-16 (b) Normalized scores with variances in Adversarial battle in scale 16-16 (c) Normalized scores with variances in Food Collection with 24 agents
We first report the normalized score in Fig. 13 by comparing all the methods against EPC. EPC is consistently better than all methods in all the scales.
Besides, we also report the raw reward numbers when competing against EPC. Since the Predator-prey game is a zero-sum game, we simply report the predator rewards (the prey reward is exactly the negative value).
scale MADDPG mean field Att-MADDPG vanilla-PC EPC 3-2 10.405 7.39 8.675 8.675 12.662 6-4 31.458 25.529 31.747 45.155 54.546
12-8 74.939 64.377 133.261 200.638 214.328
Furthermore, we also demonstrate the results of full pairwise competition between every two methods for scale 12-8 below. Consistently, we can see that EPC predators always have the highest scores as in the last row. When competing against EPC prey, the lowest rewards are observed.
predator prey MADDPG mean-field Att-MADDPG Vanilla-PC EPC
MADDPG 229.887 249.4 247.423 122.878 74.939 mean-field 207.022 210.838 228.469 107.811 64.377
Att-MADDPG 569.862 532.611 373.979 204.743 133.261 Vanilla-PC 758.293 737.067 521.486 303.86 200.638
EPC 827.298 764.417 519.319 299.505 214.328

title: SCALING MULTI-AGENT REINFORCEMENT LEARNING