title: Taming LLMs by Scaling Learning Rates with Gradient Grouping
summary: Training large language models (LLMs) poses challenges due to their massive
  scale and heterogeneous architectures. While adaptive optimizers like AdamW
  help address gradient variations, they still struggle with efficient and
  effective parameter-wise learning rate estimation, resulting in training
  instability, slow convergence, and poor compatibility with parameter-efficient
  fine-tuning (PEFT) techniques. This work introduces Scaling with Gradient
  Grouping (SGG), an optimizer wrapper that improves adaptive learning rate
  estimation by dynamic grouping and group-specific scaling. SGG first groups
  gradient statistics in each layer into clusters and then applies
  cluster-specific scaling to calibrate learning rates for each parameter, thus
  imposing collective group-wise constraints while maintaining precise
  per-parameter adaptation. Experiments on diverse (M)LLM benchmarks show that
  SGG integrates seamlessly with existing optimizers, and offers consistent gains
  and faster convergence over baselines, with various model sizes. Its stability
  across varying batch sizes and learning rates establishes SGG as a robust
  choice for LLM optimization.
paper_authors._id: 683e5b9a1167d9630159b27f
paper_authors.name: Siyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:46:39.296Z
paper_authors._id: 640f7083208821a59b74c757
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678735253848-640f7083208821a59b74c757.jpeg
paper_authors.fullname: Siyuan Li
paper_authors.type: user
paper_authors.user: Lupin1998
paper_authors._id: 683e5b9a1167d9630159b280
paper_authors.name: Juanxi Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T09:40:52.165Z
paper_authors._id: 670880950e79a8b46f7ff9dd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/670880950e79a8b46f7ff9dd/hA1TLhwlQblkFsq8wLrkB.jpeg
paper_authors.fullname: Juanxi Tian
paper_authors.type: user
paper_authors.user: Juanxi
paper_authors._id: 683e5b9a1167d9630159b281
paper_authors.name: Zedong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:46:41.911Z
paper_authors._id: 6594d390674349122ce6f368
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6594d390674349122ce6f368/KdWz6lZyGYQpjAgBDeiC1.jpeg
paper_authors.fullname: Zedong Wang (Jacky)
paper_authors.type: user
paper_authors.user: ZedongWangAI
paper_authors._id: 683e5b9a1167d9630159b282
paper_authors.name: Xin Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:11.830Z
paper_authors._id: 66aa39349238d9c3a1c7f9dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66aa39349238d9c3a1c7f9dc/mj6r7uxEYXM502x296UMf.jpeg
paper_authors.fullname: Jin
paper_authors.type: user
paper_authors.user: Xin1118
paper_authors._id: 683e5b9a1167d9630159b283
paper_authors.name: Zicheng Liu
paper_authors._id: 683e5b9a1167d9630159b284
paper_authors.name: Wentao Zhang
paper_authors._id: 683e5b9a1167d9630159b285
paper_authors.name: Dan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-04T18:57:21.130Z
paper_authors._id: 66feab48651e00e22f33222e
paper_authors.avatarUrl: /avatars/7344377e2c796c7ec85194bb2fc78521.svg
paper_authors.fullname: Dan Xu
paper_authors.type: user
paper_authors.user: danxuhk
paper_title: Taming LLMs by Scaling Learning Rates with Gradient Grouping
paper_summary: Training large language models (LLMs) poses challenges due to their massive
  scale and heterogeneous architectures. While adaptive optimizers like AdamW
  help address gradient variations, they still struggle with efficient and
  effective parameter-wise learning rate estimation, resulting in training
  instability, slow convergence, and poor compatibility with parameter-efficient
  fine-tuning (PEFT) techniques. This work introduces Scaling with Gradient
  Grouping (SGG), an optimizer wrapper that improves adaptive learning rate
  estimation by dynamic grouping and group-specific scaling. SGG first groups
  gradient statistics in each layer into clusters and then applies
  cluster-specific scaling to calibrate learning rates for each parameter, thus
  imposing collective group-wise constraints while maintaining precise
  per-parameter adaptation. Experiments on diverse (M)LLM benchmarks show that
  SGG integrates seamlessly with existing optimizers, and offers consistent gains
  and faster convergence over baselines, with various model sizes. Its stability
  across varying batch sizes and learning rates establishes SGG as a robust
  choice for LLM optimization.
paper_ai_summary: SGG, an optimizer wrapper, enhances adaptive learning rates for large language models by grouping gradients and applying cluster-specific scaling, improving convergence and stability.
paper_ai_keywords: large language models adaptive optimizers AdamW parameter-wise learning rate estimation training instability parameter-efficient fine-tuning Scaling with Gradient Grouping gradient grouping cluster-specific scaling LLM benchmarks robust choice for LLM optimization