MegaScale: Scaling Large Language Model Training to More Than 10,000
  GPUs
We present the design, implementation and engineering experience in building
and deploying MegaScale, a production system for training large language models
(LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale
brings unprecedented challenges to training efficiency and stability. We take a
full-stack approach that co-designs the algorithmic and system components
across model block and optimizer design, computation and communication
overlapping, operator optimization, data pipeline, and network performance
tuning. Maintaining high efficiency throughout the training process (i.e.,
stability) is an important consideration in production given the long extent of
LLM training jobs. Many hard stability issues only emerge at large scale, and
in-depth observability is the key to address them. We develop a set of
diagnosis tools to monitor system components and events deep in the stack,
identify root causes, and derive effective techniques to achieve fault
tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs
Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the
MFU by 1.34x compared to Megatron-LM. We share our operational experience in
identifying and fixing failures and stragglers. We hope by articulating the
problems and sharing our experience from a systems perspective, this work can
inspire future LLM systems research.
65dd6c8aea1a202d3cc84c8b
Ziheng Jiang
65dd6c8aea1a202d3cc84c8c
Haibin Lin
65dd6c8aea1a202d3cc84c8d
Yinmin Zhong
admin_assigned
2024-02-27T09:41:08.213Z
64e5d04767a759dd7155a445
/avatars/70090182b71db4de31fb8c325d25e73b.svg
Yinmin Zhong
user
PKUFlyingPig
65dd6c8aea1a202d3cc84c8e
Qi Huang
65dd6c8aea1a202d3cc84c8f
Yangrui Chen
65dd6c8aea1a202d3cc84c90
Zhi Zhang
65dd6c8aea1a202d3cc84c91
Yanghua Peng
65dd6c8aea1a202d3cc84c92
Xiang Li
65dd6c8aea1a202d3cc84c93
Cong Xie
65dd6c8aea1a202d3cc84c94
Shibiao Nong
65dd6c8aea1a202d3cc84c95
Yulu Jia
65dd6c8aea1a202d3cc84c96
Sun He
65dd6c8aea1a202d3cc84c97
Hongmin Chen
65dd6c8aea1a202d3cc84c98
Zhihao Bai
65dd6c8aea1a202d3cc84c99
Qi Hou
65dd6c8aea1a202d3cc84c9a
Shipeng Yan
65dd6c8aea1a202d3cc84c9b
Ding Zhou
65dd6c8aea1a202d3cc84c9c
Yiyao Sheng
65dd6c8aea1a202d3cc84c9d
Zhuo Jiang
65dd6c8aea1a202d3cc84c9e
Haohan Xu
65dd6c8aea1a202d3cc84c9f
Haoran Wei
65dd6c8aea1a202d3cc84ca0
Zhang Zhang
65dd6c8aea1a202d3cc84ca1
Pengfei Nie
65dd6c8aea1a202d3cc84ca2
Leqi Zou
65dd6c8aea1a202d3cc84ca3
Sida Zhao
65dd6c8aea1a202d3cc84ca4
Liang Xiang
65dd6c8aea1a202d3cc84ca5
Zherui Liu
65dd6c8aea1a202d3cc84ca6
Zhe Li
65dd6c8aea1a202d3cc84ca7
Xiaoying Jia
65dd6c8aea1a202d3cc84ca8
Jianxi Ye
65dd6c8aea1a202d3cc84ca9
Xin Jin
65dd6c8aea1a202d3cc84caa
Xin Liu
MegaScale: Scaling Large Language Model Training to More Than 10,000
  GPUs
We present the design, implementation and engineering experience in building
and deploying MegaScale, a production system for training large language models
(LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale
brings unprecedented challenges to training efficiency and stability. We take a
full-stack approach that co-designs the algorithmic and system components
across model block and optimizer design, computation and communication
overlapping, operator optimization, data pipeline, and network performance
tuning. Maintaining high efficiency throughout the training process (i.e.,
stability) is an important consideration in production given the long extent of
LLM training jobs. Many hard stability issues only emerge at large scale, and
in-depth observability is the key to address them. We develop a set of
diagnosis tools to monitor system components and events deep in the stack,
identify root causes, and derive effective techniques to achieve fault
tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs
Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the
MFU by 1.34x compared to Megatron-LM. We share our operational experience in
identifying and fixing failures and stragglers. We hope by articulating the
problems and sharing our experience from a systems perspective, this work can
inspire future LLM systems research.
MegaScale is a production system designed to train large language models at scale using more than 10,000 GPUs, focusing on efficiency, stability, and fault tolerance through advanced observability and diagnostic tools.
MegaScale large language models (LLMs) Model FLOPs Utilization (MFU) Megatron-LM fault tolerance stragglers operator optimization data pipeline network performance tuning