title: MegaScale: Scaling Large Language Model Training to More Than 10,000
    GPUs
summary: We present the design, implementation and engineering experience in building
  and deploying MegaScale, a production system for training large language models
  (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale
  brings unprecedented challenges to training efficiency and stability. We take a
  full-stack approach that co-designs the algorithmic and system components
  across model block and optimizer design, computation and communication
  overlapping, operator optimization, data pipeline, and network performance
  tuning. Maintaining high efficiency throughout the training process (i.e.,
  stability) is an important consideration in production given the long extent of
  LLM training jobs. Many hard stability issues only emerge at large scale, and
  in-depth observability is the key to address them. We develop a set of
  diagnosis tools to monitor system components and events deep in the stack,
  identify root causes, and derive effective techniques to achieve fault
  tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs
  Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the
  MFU by 1.34x compared to Megatron-LM. We share our operational experience in
  identifying and fixing failures and stragglers. We hope by articulating the
  problems and sharing our experience from a systems perspective, this work can
  inspire future LLM systems research.
paper_authors._id: 65dd6c8aea1a202d3cc84c8b
paper_authors.name: Ziheng Jiang
paper_authors._id: 65dd6c8aea1a202d3cc84c8c
paper_authors.name: Haibin Lin
paper_authors._id: 65dd6c8aea1a202d3cc84c8d
paper_authors.name: Yinmin Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-27T09:41:08.213Z
paper_authors._id: 64e5d04767a759dd7155a445
paper_authors.avatarUrl: /avatars/70090182b71db4de31fb8c325d25e73b.svg
paper_authors.fullname: Yinmin Zhong
paper_authors.type: user
paper_authors.user: PKUFlyingPig
paper_authors._id: 65dd6c8aea1a202d3cc84c8e
paper_authors.name: Qi Huang
paper_authors._id: 65dd6c8aea1a202d3cc84c8f
paper_authors.name: Yangrui Chen
paper_authors._id: 65dd6c8aea1a202d3cc84c90
paper_authors.name: Zhi Zhang
paper_authors._id: 65dd6c8aea1a202d3cc84c91
paper_authors.name: Yanghua Peng
paper_authors._id: 65dd6c8aea1a202d3cc84c92
paper_authors.name: Xiang Li
paper_authors._id: 65dd6c8aea1a202d3cc84c93
paper_authors.name: Cong Xie
paper_authors._id: 65dd6c8aea1a202d3cc84c94
paper_authors.name: Shibiao Nong
paper_authors._id: 65dd6c8aea1a202d3cc84c95
paper_authors.name: Yulu Jia
paper_authors._id: 65dd6c8aea1a202d3cc84c96
paper_authors.name: Sun He
paper_authors._id: 65dd6c8aea1a202d3cc84c97
paper_authors.name: Hongmin Chen
paper_authors._id: 65dd6c8aea1a202d3cc84c98
paper_authors.name: Zhihao Bai
paper_authors._id: 65dd6c8aea1a202d3cc84c99
paper_authors.name: Qi Hou
paper_authors._id: 65dd6c8aea1a202d3cc84c9a
paper_authors.name: Shipeng Yan
paper_authors._id: 65dd6c8aea1a202d3cc84c9b
paper_authors.name: Ding Zhou
paper_authors._id: 65dd6c8aea1a202d3cc84c9c
paper_authors.name: Yiyao Sheng
paper_authors._id: 65dd6c8aea1a202d3cc84c9d
paper_authors.name: Zhuo Jiang
paper_authors._id: 65dd6c8aea1a202d3cc84c9e
paper_authors.name: Haohan Xu
paper_authors._id: 65dd6c8aea1a202d3cc84c9f
paper_authors.name: Haoran Wei
paper_authors._id: 65dd6c8aea1a202d3cc84ca0
paper_authors.name: Zhang Zhang
paper_authors._id: 65dd6c8aea1a202d3cc84ca1
paper_authors.name: Pengfei Nie
paper_authors._id: 65dd6c8aea1a202d3cc84ca2
paper_authors.name: Leqi Zou
paper_authors._id: 65dd6c8aea1a202d3cc84ca3
paper_authors.name: Sida Zhao
paper_authors._id: 65dd6c8aea1a202d3cc84ca4
paper_authors.name: Liang Xiang
paper_authors._id: 65dd6c8aea1a202d3cc84ca5
paper_authors.name: Zherui Liu
paper_authors._id: 65dd6c8aea1a202d3cc84ca6
paper_authors.name: Zhe Li
paper_authors._id: 65dd6c8aea1a202d3cc84ca7
paper_authors.name: Xiaoying Jia
paper_authors._id: 65dd6c8aea1a202d3cc84ca8
paper_authors.name: Jianxi Ye
paper_authors._id: 65dd6c8aea1a202d3cc84ca9
paper_authors.name: Xin Jin
paper_authors._id: 65dd6c8aea1a202d3cc84caa
paper_authors.name: Xin Liu
paper_title: MegaScale: Scaling Large Language Model Training to More Than 10,000
    GPUs
paper_summary: We present the design, implementation and engineering experience in building
  and deploying MegaScale, a production system for training large language models
  (LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale
  brings unprecedented challenges to training efficiency and stability. We take a
  full-stack approach that co-designs the algorithmic and system components
  across model block and optimizer design, computation and communication
  overlapping, operator optimization, data pipeline, and network performance
  tuning. Maintaining high efficiency throughout the training process (i.e.,
  stability) is an important consideration in production given the long extent of
  LLM training jobs. Many hard stability issues only emerge at large scale, and
  in-depth observability is the key to address them. We develop a set of
  diagnosis tools to monitor system components and events deep in the stack,
  identify root causes, and derive effective techniques to achieve fault
  tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs
  Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the
  MFU by 1.34x compared to Megatron-LM. We share our operational experience in
  identifying and fixing failures and stragglers. We hope by articulating the
  problems and sharing our experience from a systems perspective, this work can
  inspire future LLM systems research.
paper_ai_keywords: MegaScale large language models (LLMs) GPUs training efficiency stability algorithmic components optimizer design computation and communication overlapping operator optimization data pipeline network performance tuning Model FLOPs Utilization (MFU) Megatron-LM stragglers fault tolerance