Published as a conference paper at ICLR 2018 LEARNING DEEP MEAN FIELD GAMES FOR MODEL- ING LARGE POPULATION BEHAVIOR
We consider the problem of representing collective behavior of large populations and predicting the evolution of a population distribution over a discrete state space. A discrete time mean field game (MFG) is motivated as an interpretable model founded on game theory for understanding the aggregate effect of individual actions and predicting the temporal evolution of population distributions. We achieve a synthesis of MFG and Markov decision processes (MDP) by showing that a special MFG is reducible to an MDP. This enables us to broaden the scope of mean field game theory and infer MFG models of large real-world systems via deep inverse reinforcement learning. Our method learns both the reward function and forward dynamics of an MFG from real data, and we report the first empirical test of a mean field game model of a real-world social media population.
This paper attacks an important problems with an interesting and promising methodology.  The authors deal with inference in models of collective behavior, specifically at how to infer the parameters of a mean field game representation of collective behavior. The technique the authors innovate is to specify a mean field game as a model, and then use inverse reinforcement learning to learn the reward functions of agents in the mean field game.
This work has many virtues, and could be an impactful piece. There is still minimal work at the intersection of machine learning and collective behavior, and this paper could help to stimulate the growth of that intersection.  The application to collective behavior could be an interesting novel application to many in machine learning, and conversely the inference techniques that are innovated should be novel to many researchers in collective behavior.
At the same time, the scientific content of the work has critical conceptual flaws.  Most fundamentally, the authors appear to implicitly center their work around highly controversial claims about the ontological status of group optimization, without the careful justification necessary to make this kind of argument.  In addition to that, the authors appear to implicitly assume that utility function inference can be used for causal inference. 
That is, there are two distinct mistakes the authors make in their scientific claims:
1) The authors write as if mean field games represent population optimization (Mean field games are not about what a _group_ optimizes; they are about what _individuals_ optimize, and this individual optimization leads to certain patterns in collective behaviors)
2) The authors write as if utility/reward function inference alone can provide causal understanding of collective or individual behavior
1 - 
I should say that I am highly sympathetic to the claim that many types of collective behavior can be viewed as optimizing some kind of objective function.  However, this claim is far from mainstream, and is in fact highly contested.  For instance, many prominent pieces of work in the study of collective behavior have highlighted its irrational aspects, from the madness of crowds to herding in financial markets.
Since it is so fringe to attribute causal agency to groups, let alone optimal agency, in the remainder of my review I will give the authors the benefit of the doubt and assume when they say things like "population behavior may be optimal", they mean "the behavior of individuals within a population may be optimal".  If the authors do mean to say this, they should be more careful about their language use in this regard (individuals are the actors, not populations).  If the authors do indeed mean to attribute causal agency to groups (as suggested in their MDP representation), they will run into all the criticisms I would have about an individual-level analysis and more.  Suffice it to say, mean field games themselves don't make claims about aggregate-level optimization.  A Nash equilibrium achieves a balance between individual-level reward functions. These reward functions are only interpretable at the individual level.  There is no objective function the group itself in aggregate is optimizing in mean field games.  For instance, even though the mean field game model of the Mexican wave produces wave solutions, the model is premised on people having individual utility functions that lead to emergent wave behavior.  The model does not have the representational capacity to explain that people actually intend to create the emergent behavior of a wave (even though in this case they do).  Furthermore, the fact that mean field games aggregate to a single-agent MDP does not imply that that the group can rightfully be thought of as an agent optimizing the reward function, because there is an exact correspondence between the rewards of the individual agents in the MFG and of the aggregate agent in the MDP by construction.
2 -
The authors also claim that their inference methods can help explain why people choose to talk about certain topics. As far as the extent to which utility / reward function inference can provide causal explanations of individual (or collective) behavior, the argument that is invariably brought against a claim of optimization is that almost any behavior can be explained as optimal post-hoc with enough degrees of freedom in the utiliy function of the behavioral model. Since optimization frameworks are so flexible, they have little explanatory power and are hard to falsify.  In fact, there is literally no way that the modeling framework of the authors even affords the possibility that individual/collective behavior is not optimal.  Optimality is taken as an assumption that allows the authors to infer what reward function is being optimized. 
The authors state that the reward function they infer helps to interpret collective behavior because it reveals what people are optimizing.  However, the reward function actually discovered is not interpretable at all. It is simply a summary of the statistical properties of changes in popularity of the topics of conversation in the Twitter data the authors' study. To quote the authors' insights: "The learned reward function reveals that a real social media population favors states characterized by a highly non-uniform distribution with negative mass gradient in decreasing order of topic popularity, as well as transitions that increase this distribution imbalance."  The authors might as well have simply visualized the topic popularities and changes in popularities to arrive at such an insight. To take the authors claims literally, we would say that people have an intrinsic preference for everyone to arbitrarily be talking about the same thing, regardless of the content or relevance of that topic.  To draw an analogy, this is like observing that on some days everybody on the street is carrying open umbrellas and on other days not, and inferring that the people on the street have a preference for everyone having their umbrellas open together (and the model would then predict that if one person opens an umbrella on a sunny day, everybody else will too).
To the authors credit, they do make a brief attempt to present empirical evidence for their optimization view, stating succinctly: "The high prediction accuracy of the learned policy provides evidence that real population behavior can be understood and modeled as the result of an emergent population-level optimization with respect to a reward function." Needless to say, this one-sentence argument for a highly controversial scientific claims falls flat on closer inspection. Setting aside the issues of correlation versus causation, predictive accuracy does not in and of itself provide scientific plausibility. When an n-gram model produces text that is in the style of a particular writer, we do not conclude that the writer must have been composing based on the n-gram's generative mechanism.  Predictive accuracy only provides evidence when combined in the first place with scientific plausibility through other avenues of evidence.
The authors could attempt to address these issues by making what is called an "as-if" argument, but it's not even clear such an argument could work here in general.  
With all this in mind, it would be more instructive to show that the inference method the authors introduce could infer the correct utility functions used in standard mean field games, such as modeling traffic congestion and the Mexican wave.  
--
All that said, the general approach taken in the authors' work is highly promising, and there are many fruitful directions I would be exicted to see this work taken --- e.g., combining endogenous and exogenous rewards or looking at more complex applications.  As a technical contribution, the paper is wonderful, and I would enthusiastically support acceptance.  The authors simply either need to be much more careful with the scientific claims about collective behavior they make, or limit the scope of the contribution of the paper to be modeling / inference  in the area of collective behavior.  Mean field games are an important class of models in collective behavior, and being able to infer their parameters is a nice step forward purely due to the importance of that class of games.  Identifying where the authors' inference method could be applied to draw valid scientific conclusions about collective behavior could then be an avenue for future work.  Examples of plausible scientific applications might include parameter inference in settings where mean field games are already typically applied in order to improve the fit of those models or to learn about trade-offs people make in their utility functions in those settings.
--
Other minor comments:
- (Introduction) It is not clear at all how the Arab Spring, Black Lives Matter, and fake news are similar --- i.e., whether a single model could provide insight into these highly heterogeneous events --- nor is it clear what end the authors hope to achieve by modeling them --- the ethics of modeling protests in a field crowded with powerful institutional actors is worth carefully considering.
- If I understand correctly, the fact that the authors assume a factored reward function seems limiting. Isn't the major benefit of game theory it's ability to accommodate utility functions that depend on the actions of others?
- The authors state that one of their essential insights is that "solving the optimization problem of a single-agent MDP is equivalent to solving the inference problem of an MFG." This statement feels a bit too cute at the expense of clarity. The authors perform inference via inverse-RL, so it's more clear to say the authors are attempting to use statistical inference to figure out what is being optimized.
- The relationship between MFGs and a single-agent MDP is nice and a fine observation, but not as surprising as the authors frame it as. Any multiagent MDP can be naively represented as a single-agent MDP where the agent has control over the entire population, and we already know that stochastic games are closely related to MDPs.  It's therefore hard to imagine that there woudn't be some sort of correspondence.