  Throughout scientific history, overarching theoretical frameworks have
allowed researchers to grow beyond personal intuitions and culturally biased
theories. They allow to verify and replicate existing findings, and to link is
connected results. The notion of self-play, albeit often cited in multiagent
Reinforcement Learning, has never been grounded in a formal model. We present a
formalized framework, with clearly defined assumptions, which encapsulates the
meaning of self-play as abstracted from various existing self-play algorithms.
This framework is framed as an approximation to a theoretical solution concept
for multiagent training. On a simple environment, we qualitatively measure how
well a subset of the captured self-play methods approximate this solution when
paired with the famous PPO algorithm. We also provide insights on interpreting
quantitative metrics of performance for self-play training. Our results
indicate that, throughout training, various self-play definitions exhibit
cyclic policy evolutions.

Artificial Intelligence;Computer Science and Game Theory