Boosting Large-scale Parallel Training Efficiency with C4: A
  Communication-Driven Approach
The emergence of Large Language Models (LLMs) has necessitated the adoption
of parallel training techniques, involving the deployment of thousands of GPUs
to train a single model. Unfortunately, we have found that the efficiency of
current parallel training is often suboptimal, largely due to the following two
main issues. Firstly, hardware failures are inevitable, leading to
interruptions in the training tasks. The inability to quickly identify the
faulty components results in a substantial waste of GPU resources. Secondly,
since GPUs must wait for parameter synchronization to complete before
proceeding to the next round of computation, network congestions can greatly
increase the waiting time for GPUs. To address these challenges, this paper
introduces a communication-driven solution, namely the C4. The key insights of
C4 are two folds. First, in parallel training, collective communication
exhibits periodic and homogeneous characteristics, so any anomalies are
certainly due to some form of hardware malfunction. By leveraging this feature,
C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
restart the task, thereby avoiding resource wastage caused by delays in anomaly
detection. Second, the predictable communication model of collective
communication, involving few large flows, allows C4 to efficiently execute
traffic planning, substantially reducing network congestion. C4 has been
extensively implemented across our production systems, cutting error-induced
overhead by roughly 30% and enhancing runtime performance by about 15% for
certain applications with moderate communication costs.
66666383113312a4a955f263
Jianbo Dong
66666383113312a4a955f264
Bin Luo
66666383113312a4a955f265
Jun Zhang
66666383113312a4a955f266
Pengcheng Zhang
66666383113312a4a955f267
Fei Feng
66666383113312a4a955f268
Yikai Zhu
66666383113312a4a955f269
Ang Liu
66666383113312a4a955f26a
Zian Chen
66666383113312a4a955f26b
Yi Shi
66666383113312a4a955f26c
Hairong Jiao
66666383113312a4a955f26d
Gang Lu
66666383113312a4a955f26e
Yu Guan
66666383113312a4a955f26f
Ennan Zhai
66666383113312a4a955f270
Wencong Xiao
66666383113312a4a955f271
Hanyu Zhao
66666383113312a4a955f272
Man Yuan
66666383113312a4a955f273
Siran Yang
66666383113312a4a955f274
Xiang Li
66666383113312a4a955f275
Jiamang Wang
66666383113312a4a955f276
Rui Men
claimed_verified
2024-09-23T16:30:02.460Z
6209bb200436d7d6f27cbeea
/avatars/0b8a72a8b66ef7b36780fe2ccc343f78.svg
Iurnem
user
Iurnem
66666383113312a4a955f277
Jianwei Zhang
66666383113312a4a955f278
Huang Zhong
66666383113312a4a955f279
Dennis Cai
66666383113312a4a955f27a
Yuan Xie
66666383113312a4a955f27b
Binzhang Fu
Boosting Large-scale Parallel Training Efficiency with C4: A
  Communication-Driven Approach
The emergence of Large Language Models (LLMs) has necessitated the adoption
of parallel training techniques, involving the deployment of thousands of GPUs
to train a single model. Unfortunately, we have found that the efficiency of
current parallel training is often suboptimal, largely due to the following two
main issues. Firstly, hardware failures are inevitable, leading to
interruptions in the training tasks. The inability to quickly identify the
faulty components results in a substantial waste of GPU resources. Secondly,
since GPUs must wait for parameter synchronization to complete before
proceeding to the next round of computation, network congestions can greatly
increase the waiting time for GPUs. To address these challenges, this paper
introduces a communication-driven solution, namely the C4. The key insights of
C4 are two folds. First, in parallel training, collective communication
exhibits periodic and homogeneous characteristics, so any anomalies are
certainly due to some form of hardware malfunction. By leveraging this feature,
C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
restart the task, thereby avoiding resource wastage caused by delays in anomaly
detection. Second, the predictable communication model of collective
communication, involving few large flows, allows C4 to efficiently execute
traffic planning, substantially reducing network congestion. C4 has been
extensively implemented across our production systems, cutting error-induced
overhead by roughly 30% and enhancing runtime performance by about 15% for
certain applications with moderate communication costs.
C4 is a communication-driven solution that improves parallel LLM training by reducing hardware failure disruptions and network congestion through efficient anomaly detection and traffic planning.
Large Language Models (LLMs) parallel training GPUs collective communication hardware failures parameter synchronization network congestion C4