Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity
<|reference_start|>Adam Exploits $\ell_\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity: Adam outperforms SGD when training language models. Yet this advantage is not well-understood theoretically -- previous convergence analysis for Adam and SGD mainly focuses on the number of steps $T$ and is already minimax-optimal in non-convex cases, which are both $\widetilde{O}(T^{-1/4})$. In this work, we argue that the exploitation of nice $\ell_\infty$-geometry is the key advantage of Adam over SGD. More specifically, we give a new convergence analysis for Adam under novel assumptions that loss is smooth under $\ell_\infty$-geometry rather than the more common $\ell_2$-geometry, which yields a much better empirical smoothness constant for GPT-2 and ResNet models. Our experiments confirm that Adam performs much worse when the favorable $\ell_\infty$-geometry is changed while SGD provably remains unaffected. We also extend the convergence analysis to blockwise Adam under novel blockwise smoothness assumptions.<|reference_end|>
@article{xie2024adam,
    title={Adam Exploits $\ell_\infty$-geometry of Loss Landscape via
  Coordinate-wise Adaptivity},
    author={Shuo Xie, Mohamad Amin Mohamadi, Zhiyuan Li},
    journal={arXiv preprint arXiv:2410.08198},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.08198},
    primaryClass={cs.LG}
}