The Danger Of Arrogance: Welfare Equilibra As A Solution To Stackelberg Self-Play In Non-Coincidental Games
<|reference_start|>The Danger Of Arrogance: Welfare Equilibra As A Solution To Stackelberg Self-Play In Non-Coincidental Games: The increasing prevalence of multi-agent learning systems in society necessitates understanding how to learn effective and safe policies in general-sum multi-agent environments against a variety of opponents, including self-play. General-sum learning is difficult because of non-stationary opponents and misaligned incentives. Our first main contribution is to show that many recent approaches to general-sum learning can be derived as approximations to Stackelberg strategies, which suggests a framework for developing new multi-agent learning algorithms. We then define non-coincidental games as games in which the Stackelberg strategy profile is not a Nash Equilibrium. This notably includes several canonical matrix games and provides a normative theory for why existing algorithms fail in self-play in such games. We address this problem by introducing Welfare Equilibria (WE) as a generalisation of Stackelberg Strategies, which can recover desirable Nash Equilibria even in non-coincidental games. Finally, we introduce Welfare Function Search (WelFuSe) as a practical approach to finding desirable WE against unknown opponents, which finds more mutually desirable solutions in self-play, while preserving performance against naive learning opponents.<|reference_end|>
@article{levi2024the,
    title={The Danger Of Arrogance: Welfare Equilibra As A Solution To Stackelberg
  Self-Play In Non-Coincidental Games},
    author={Jake Levi, Chris Lu, Timon Willi, Christian Schroeder de Witt, Jakob
  Foerster},
    journal={arXiv preprint arXiv:2402.01088},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.01088},
    primaryClass={cs.GT cs.MA}
}