Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization
<|reference_start|>Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization: Adam with decoupled weight decay, also known as AdamW, is widely acclaimed for its superior performance in language modeling tasks, surpassing Adam with $\ell_2$ regularization in terms of generalization and optimization. However, this advantage is not theoretically well-understood. One challenge here is that though intuitively Adam with $\ell_2$ regularization optimizes the $\ell_2$ regularized loss, it is not clear if AdamW optimizes a specific objective. In this work, we make progress toward understanding the benefit of AdamW by showing that it implicitly performs constrained optimization. More concretely, we show in the full-batch setting, if AdamW converges with any non-increasing learning rate schedule whose partial sum diverges, it must converge to a KKT point of the original loss under the constraint that the $\ell_\infty$ norm of the parameter is bounded by the inverse of the weight decay factor. This result is built on the observation that Adam can be viewed as a smoothed version of SignGD, which is the normalized steepest descent with respect to $\ell_\infty$ norm, and a surprising connection between normalized steepest descent with weight decay and Frank-Wolfe.<|reference_end|>
@article{xie2024implicit,
    title={Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization},
    author={Shuo Xie, Zhiyuan Li},
    journal={arXiv preprint arXiv:2404.04454},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.04454},
    primaryClass={cs.LG math.OC stat.ML}
}