Long-term causal effects via behavioral game theory
Planned experiments are the gold standard in reliably comparing the causal effect of switching from a baseline policy to a new policy. One critical shortcoming of classical experimental methods, however, is that they typically do not take into account the dynamic nature of response to policy changes. For instance, in an experiment where we seek to understand the effects of a new ad pricing policy on auction revenue, agents may adapt their bidding in response to the experimental pricing changes. Thus, causal effects of the new pricing policy after such adaptation period, the long-term causal effects, are not captured by the classical methodology even though they clearly are more indicative of the value of the new policy. Here, we formalize a framework to define and estimate long-term causal effects of policy changes in multiagent economies. Central to our approach is behavioral game theory, which we leverage to formulate the ignorability assumptions that are necessary for causal inference. Under such assumptions we estimate long-term causal effects through a latent space approach, where a behavioral model of how agents act conditional on their latent behaviors is combined with a temporal model of how behaviors evolve over time.
This paper addresses an issue that is common in randomized controlled trials (or A/B tests): inferences may be invalid if there is nonstationarity, and in particular if users strategically respond dynamically to the differences in the two cells.  The authors attempt to calibrate a strategic model using short-run data to then estimate a long-run causal effect.
 ---The way you refer to "causal effect" and "the fundamental problem of causal inference" is somewhat nonstandard.  Typically (in the Rubin potential outcomes model, which is what you are building on), the causal effect is defined at the individual level, with a "treatment" outcome and "control" outcome for each experimental unit.  The fundamental problem of causal inference is that only one of these two outcomes is actually observed for each experimental unit.  You seem to be focusing on a slightly different issue, which is that the effect of treating the entire population cannot be determined correctly from just data when half the population is treated.  It seems to me that this issue -- which can arise due to a variety of violations of the SUTVA assumption -- can exist independent of whether there is a multiagent interaction.  Conversely, it seems multiagent considerations are relevant even when defining causal effects at the sub-population level.  ---Your references are somewhat limited to recent work.  There has been extensive work on causal inference in nonstationary environments, particularly by econometricians and time series analysts.  It would be good to situate your work better relative to this work.  Your main difference lies in the particular structure of behavioral model you assume, and I judged the paper on this basis.   ----A significant limitation that I see is that the modeling approach requires games and behaviors to be defined before the estimation procedure can be carried out.  One could, instead, imagine a fully nonparametric approach to causal inference that just aimed to remove any nonstationarity in treatment and control before estimating a treatment effect.  The approach you take requires that one should believe constructing the necessary spaces of games and behaviors would be both reasonably specificed, and computationally tractable.  ---The main contribution is in the modeling, as the theorem follows naturally from the modeling assumptions.  The theorem is making an asymptotic statement imprecisely; it would be better to formally state the result as well.  ---The numerics are carried out for a relatively simplified two player game, whereas the motivating examples are all drawn from ad auctions.  I would like to understand whether this approach can be reasonably used in a more practical setting.  For example, in an ad auction setting, how would you restrict the behaviors that you consider?  As the paper currently stands, the example is a bit too simple, considering the significant model complexity that would arise if one tried to capture all behaviors possible in more complex (and real-world) strategic environments.  ---Overall, I thought the authors took an intriguing approach to a very difficult problem.  I was generally left with the impression that the work is promising but still preliminary, and would benefit from some deeper investigation of the applicability of the methods, as well as comparison to the very extensive related literature.