title: Unicron: Economizing Self-Healing LLM Training at Scale
summary: Training large-scale language models is increasingly critical in various
  domains, but it is hindered by frequent failures, leading to significant time
  and economic costs. Current failure recovery methods in cloud-based settings
  inadequately address the diverse and complex scenarios that arise, focusing
  narrowly on erasing downtime for individual tasks without considering the
  overall cost impact on a cluster. We introduce Unicron, a workload manager
  designed for efficient self-healing in large-scale language model training.
  Unicron optimizes the training process by minimizing failure-related costs
  across multiple concurrent tasks within a cluster. Its key features include
  in-band error detection for real-time error identification without extra
  overhead, a dynamic cost-aware plan generation mechanism for optimal
  reconfiguration, and an efficient transition strategy to reduce downtime during
  state changes. Deployed on a 128-GPU distributed cluster, Unicron demonstrates
  up to a 1.9x improvement in training efficiency over state-of-the-art methods,
  significantly reducing failure recovery costs and enhancing the reliability of
  large-scale language model training.
paper_authors._id: 65938aa435c41262d6ebd23a
paper_authors.name: Tao He
paper_authors._id: 65938aa435c41262d6ebd23b
paper_authors.name: Xue Li
paper_authors._id: 65938aa435c41262d6ebd23c
paper_authors.name: Zhibin Wang
paper_authors._id: 65938aa435c41262d6ebd23d
paper_authors.name: Kun Qian
paper_authors._id: 65938aa435c41262d6ebd23e
paper_authors.name: Jingbo Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:34:41.282Z
paper_authors._id: 643f4e286d5be535d28200e7
paper_authors.avatarUrl: /avatars/046d3db478e16dde7ff8a66ce83d90f5.svg
paper_authors.fullname: Jingbo Xu
paper_authors.type: user
paper_authors.user: yecol
paper_authors._id: 65938aa435c41262d6ebd23f
paper_authors.name: Wenyuan Yu
paper_authors._id: 65938aa435c41262d6ebd240
paper_authors.name: Jingren Zhou
paper_title: Unicron: Economizing Self-Healing LLM Training at Scale
paper_summary: Training large-scale language models is increasingly critical in various
  domains, but it is hindered by frequent failures, leading to significant time
  and economic costs. Current failure recovery methods in cloud-based settings
  inadequately address the diverse and complex scenarios that arise, focusing
  narrowly on erasing downtime for individual tasks without considering the
  overall cost impact on a cluster. We introduce Unicron, a workload manager
  designed for efficient self-healing in large-scale language model training.
  Unicron optimizes the training process by minimizing failure-related costs
  across multiple concurrent tasks within a cluster. Its key features include
  in-band error detection for real-time error identification without extra
  overhead, a dynamic cost-aware plan generation mechanism for optimal
  reconfiguration, and an efficient transition strategy to reduce downtime during
  state changes. Deployed on a 128-GPU distributed cluster, Unicron demonstrates
  up to a 1.9x improvement in training efficiency over state-of-the-art methods,
  significantly reducing failure recovery costs and enhancing the reliability of
  large-scale language model training.
paper_ai_summary: Unicron, a workload manager, enhances the self-healing and efficiency of large-scale language model training by optimizing for cost and minimizing downtime across multiple tasks.
paper_ai_keywords: workload manager error detection cost-aware plan generation state changes distributed cluster training efficiency reliability