title: Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
summary: This paper comprehensively evaluates several recently proposed optimizers for
  4-bit training, revealing that low-bit precision amplifies sensitivity to
  learning rates and often causes unstable gradient norms, leading to divergence
  at higher learning rates. Among these, SPAM, a recent optimizer featuring
  momentum reset and spike-aware gradient clipping, achieves the best performance
  across various bit levels, but struggles to stabilize gradient norms, requiring
  careful learning rate tuning. To address these limitations, we propose
  Stable-SPAM, which incorporates enhanced gradient normalization and clipping
  techniques. In particular, Stable-SPAM (1) adaptively updates the clipping
  threshold for spiked gradients by tracking their historical maxima; (2)
  normalizes the entire gradient matrix based on its historical l_2-norm
  statistics; and (3) inherits momentum reset from SPAM to periodically reset
  the first and second moments of Adam, mitigating the accumulation of spiked
  gradients. Extensive experiments show that Stable-SPAM effectively stabilizes
  gradient norms in 4-bit LLM training, delivering superior performance compared
  to Adam and SPAM. Notably, our 4-bit LLaMA-1B model trained with Stable-SPAM
  outperforms the BF16 LLaMA-1B trained with Adam by up to 2 perplexity.
  Furthermore, when both models are trained in 4-bit, Stable-SPAM achieves the
  same loss as Adam while requiring only about half the training steps. Code is
  available at https://github.com/TianjinYellow/StableSPAM.git.
paper_authors._id: 67bd9b40478ef7c36240c6e6
paper_authors.name: Tianjin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:30:14.371Z
paper_authors._id: 64cd4743a785f2043b32915e
paper_authors.avatarUrl: /avatars/ba0b497a194dfea8449112d71fc67654.svg
paper_authors.fullname: Tianjin Huang
paper_authors.type: user
paper_authors.user: TianjinHuang
paper_authors._id: 67bd9b40478ef7c36240c6e7
paper_authors.name: Haotian Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:30:36.869Z
paper_authors._id: 67577b39a6073ee33f97cdd9
paper_authors.avatarUrl: /avatars/c05788ebc7c59b13fcddf8ff88540f79.svg
paper_authors.fullname: Haotian Hu
paper_authors.type: user
paper_authors.user: cspikachu
paper_authors._id: 67bd9b40478ef7c36240c6e8
paper_authors.name: Zhenyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:30:38.930Z
paper_authors._id: 649c888f67fd6c6aa97e5f85
paper_authors.avatarUrl: /avatars/9967b729916d1128773102797fed1673.svg
paper_authors.fullname: Zhenyu Zhang
paper_authors.type: user
paper_authors.user: Kyriection
paper_authors._id: 67bd9b40478ef7c36240c6e9
paper_authors.name: Gaojie Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T16:30:59.295Z
paper_authors._id: 6662bd6b6016f6effa6ce492
paper_authors.avatarUrl: /avatars/c8e6b4b4a64f7ca0ddc91af5217a791b.svg
paper_authors.fullname: gaojie jin
paper_authors.type: user
paper_authors.user: sggjin
paper_authors._id: 67bd9b40478ef7c36240c6ea
paper_authors.name: Xiang Li
paper_authors._id: 67bd9b40478ef7c36240c6eb
paper_authors.name: Li Shen
paper_authors._id: 67bd9b40478ef7c36240c6ec
paper_authors.name: Tianlong Chen
paper_authors._id: 67bd9b40478ef7c36240c6ed
paper_authors.name: Lu Liu
paper_authors._id: 67bd9b40478ef7c36240c6ee
paper_authors.name: Qingsong Wen
paper_authors._id: 67bd9b40478ef7c36240c6ef
paper_authors.name: Zhangyang Wang
paper_authors._id: 67bd9b40478ef7c36240c6f0
paper_authors.name: Shiwei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T11:14:48.533Z
paper_authors._id: 65b04d2291e63920a7898c9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Shiweiliuiiiiiii
paper_title: Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam
paper_summary: This paper comprehensively evaluates several recently proposed optimizers for
  4-bit training, revealing that low-bit precision amplifies sensitivity to
  learning rates and often causes unstable gradient norms, leading to divergence
  at higher learning rates. Among these, SPAM, a recent optimizer featuring
  momentum reset and spike-aware gradient clipping, achieves the best performance
  across various bit levels, but struggles to stabilize gradient norms, requiring
  careful learning rate tuning. To address these limitations, we propose
  Stable-SPAM, which incorporates enhanced gradient normalization and clipping
  techniques. In particular, Stable-SPAM (1) adaptively updates the clipping
  threshold for spiked gradients by tracking their historical maxima; (2)
  normalizes the entire gradient matrix based on its historical l_2-norm
  statistics; and (3) inherits momentum reset from SPAM to periodically reset
  the first and second moments of Adam, mitigating the accumulation of spiked
  gradients. Extensive experiments show that Stable-SPAM effectively stabilizes
  gradient norms in 4-bit LLM training, delivering superior performance compared
  to Adam and SPAM. Notably, our 4-bit LLaMA-1B model trained with Stable-SPAM
  outperforms the BF16 LLaMA-1B trained with Adam by up to 2 perplexity.
  Furthermore, when both models are trained in 4-bit, Stable-SPAM achieves the
  same loss as Adam while requiring only about half the training steps. Code is
  available at https://github.com/TianjinYellow/StableSPAM.git.
paper_ai_summary: Stable-SPAM, an enhanced version of SPAM optimizer, stabilizes gradient norms and improves performance in low-bit precision training across various bit levels.
paper_ai_keywords: 4-bit training low-bit precision learning rates gradient norms divergence SPAM momentum reset spike-aware gradient clipping Stable-SPAM adaptive clipping threshold historical maxima $l_2$-norm statistics Adam LLM training LLaMA-1B perplexity training steps