Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed
<|reference_start|>Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed: Methods with adaptive stepsizes, such as AdaGrad and Adam, are essential for training modern Deep Learning models, especially Large Language Models. Typically, the noise in the stochastic gradients is heavy-tailed for the later ones. Gradient clipping provably helps to achieve good high-probability convergence for such noises. However, despite the similarity between AdaGrad/Adam and Clip-SGD, the high-probability convergence of AdaGrad/Adam has not been studied in this case. In this work, we prove that AdaGrad (and its delayed version) can have provably bad high-probability convergence if the noise is heavy-tailed. To fix this issue, we propose a new version of AdaGrad called Clip-RAdaGradD (Clipped Reweighted AdaGrad with Delay) and prove its high-probability convergence bounds with polylogarithmic dependence on the confidence level for smooth convex/non-convex stochastic optimization with heavy-tailed noise. Our empirical evaluations, including NLP model fine-tuning, highlight the superiority of clipped versions of AdaGrad/Adam in handling the heavy-tailed noise.<|reference_end|>
@article{chezhegov2024gradient,
    title={Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed},
    author={Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov, Aleksandr
  Beznosikov, Alexander Gasnikov, Samuel Horv'ath, Martin Tak'av{c}, Eduard
  Gorbunov},
    journal={arXiv preprint arXiv:2406.04443},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.04443},
    primaryClass={cs.LG math.OC}
}