Fast Last-Iterate Convergence of Learning in Games Requires Forgetful Algorithms
<|reference_start|>Fast Last-Iterate Convergence of Learning in Games Requires Forgetful Algorithms: Self-play via online learning is one of the premier ways to solve large-scale two-player zero-sum games, both in theory and practice. Particularly popular algorithms include optimistic multiplicative weights update (OMWU) and optimistic gradient-descent-ascent (OGDA). While both algorithms enjoy $O(1/T)$ ergodic convergence to Nash equilibrium in two-player zero-sum games, OMWU offers several advantages including logarithmic dependence on the size of the payoff matrix and $\widetilde{O}(1/T)$ convergence to coarse correlated equilibria even in general-sum games. However, in terms of last-iterate convergence in two-player zero-sum games, an increasingly popular topic in this area, OGDA guarantees that the duality gap shrinks at a rate of $O(1/\sqrt{T})$, while the best existing last-iterate convergence for OMWU depends on some game-dependent constant that could be arbitrarily large. This begs the question: is this potentially slow last-iterate convergence an inherent disadvantage of OMWU, or is the current analysis too loose? Somewhat surprisingly, we show that the former is true. More generally, we prove that a broad class of algorithms that do not forget the past quickly all suffer the same issue: for any arbitrarily small $\delta>0$, there exists a $2\times 2$ matrix game such that the algorithm admits a constant duality gap even after $1/\delta$ rounds. This class of algorithms includes OMWU and other standard optimistic follow-the-regularized-leader algorithms.<|reference_end|>
@article{cai2024fast,
    title={Fast Last-Iterate Convergence of Learning in Games Requires Forgetful
  Algorithms},
    author={Yang Cai, Gabriele Farina, Julien Grand-Cl'ement, Christian Kroer,
  Chung-Wei Lee, Haipeng Luo, Weiqiang Zheng},
    journal={arXiv preprint arXiv:2406.10631},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.10631},
    primaryClass={cs.GT cs.LG math.OC}
}