title: ZClip: Adaptive Spike Mitigation for LLM Pre-Training
summary: Training large language models (LLMs) presents numerous challenges, including
  gradient instability and loss spikes. These phenomena can lead to catastrophic
  divergence, requiring costly checkpoint restoration and data batch skipping.
  Traditional gradient clipping techniques, such as constant or norm-based
  methods, fail to address these issues effectively due to their reliance on
  fixed thresholds or heuristics, leading to inefficient learning and requiring
  frequent manual intervention. In this work, we propose ZClip, an adaptive
  gradient clipping algorithm that dynamically adjusts the clipping threshold
  based on statistical properties of gradient norms over time. Unlike prior
  reactive strategies, ZClip proactively adapts to training dynamics without
  making any prior assumptions on the scale and the temporal evolution of
  gradient norms. At its core, it leverages z-score-based anomaly detection to
  identify and mitigate large gradient spikes, preventing malignant loss spikes
  while not interfering with convergence otherwise. Our code is available at:
  https://github.com/bluorion-com/ZClip.
paper_authors._id: 67ef5a3d4417508df8d99dad
paper_authors.name: Abhay Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:18.512Z
paper_authors._id: 62cd4b03c5cc157be82f0b56
paper_authors.avatarUrl: /avatars/351e963c1c763d507ae78cbcd62966a3.svg
paper_authors.fullname: Abhay kumar
paper_authors.type: user
paper_authors.user: akanyaani
paper_authors._id: 67ef5a3d4417508df8d99dae
paper_authors.name: Louis Owen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:21.051Z
paper_authors._id: 6071c4b270e11b30cfcfd7a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6071c4b270e11b30cfcfd7a3/-1ekCBzSTpqxkkul0bgmI.jpeg
paper_authors.fullname: Louis Owen
paper_authors.type: user
paper_authors.user: louisowen6
paper_authors._id: 67ef5a3d4417508df8d99daf
paper_authors.name: Nilabhra Roy Chowdhury
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:12.764Z
paper_authors._id: 645a0d3dd6648853107c5fdc
paper_authors.avatarUrl: /avatars/1e3b6a4f5ce81a707ba7cbdf81631091.svg
paper_authors.fullname: Nilabhra Roy Chowdhury
paper_authors.type: user
paper_authors.user: nilabhra
paper_authors._id: 67ef5a3d4417508df8d99db0
paper_authors.name: Fabian Güra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:16.132Z
paper_authors._id: 65e4be59e8b017ee1310a1b6
paper_authors.avatarUrl: /avatars/c3f7cdf5d0859cb80bfb2b970a675dfa.svg
paper_authors.fullname: Fabian
paper_authors.type: user
paper_authors.user: gueraf
paper_title: ZClip: Adaptive Spike Mitigation for LLM Pre-Training
paper_summary: Training large language models (LLMs) presents numerous challenges, including
  gradient instability and loss spikes. These phenomena can lead to catastrophic
  divergence, requiring costly checkpoint restoration and data batch skipping.
  Traditional gradient clipping techniques, such as constant or norm-based
  methods, fail to address these issues effectively due to their reliance on
  fixed thresholds or heuristics, leading to inefficient learning and requiring
  frequent manual intervention. In this work, we propose ZClip, an adaptive
  gradient clipping algorithm that dynamically adjusts the clipping threshold
  based on statistical properties of gradient norms over time. Unlike prior
  reactive strategies, ZClip proactively adapts to training dynamics without
  making any prior assumptions on the scale and the temporal evolution of
  gradient norms. At its core, it leverages z-score-based anomaly detection to
  identify and mitigate large gradient spikes, preventing malignant loss spikes
  while not interfering with convergence otherwise. Our code is available at:
  https://github.com/bluorion-com/ZClip.
paper_ai_summary: ZClip is an adaptive gradient clipping algorithm that uses z-score-based anomaly detection to dynamically adjust clipping thresholds and prevent large gradient spikes during LLM training.
paper_ai_keywords: gradient instability loss spikes catastrophic divergence gradient clipping adaptive gradient clipping z-score-based anomaly detection