title: Boosting Large-scale Parallel Training Efficiency with C4: A
    Communication-Driven Approach
summary: The emergence of Large Language Models (LLMs) has necessitated the adoption
  of parallel training techniques, involving the deployment of thousands of GPUs
  to train a single model. Unfortunately, we have found that the efficiency of
  current parallel training is often suboptimal, largely due to the following two
  main issues. Firstly, hardware failures are inevitable, leading to
  interruptions in the training tasks. The inability to quickly identify the
  faulty components results in a substantial waste of GPU resources. Secondly,
  since GPUs must wait for parameter synchronization to complete before
  proceeding to the next round of computation, network congestions can greatly
  increase the waiting time for GPUs. To address these challenges, this paper
  introduces a communication-driven solution, namely the C4. The key insights of
  C4 are two folds. First, in parallel training, collective communication
  exhibits periodic and homogeneous characteristics, so any anomalies are
  certainly due to some form of hardware malfunction. By leveraging this feature,
  C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
  restart the task, thereby avoiding resource wastage caused by delays in anomaly
  detection. Second, the predictable communication model of collective
  communication, involving few large flows, allows C4 to efficiently execute
  traffic planning, substantially reducing network congestion. C4 has been
  extensively implemented across our production systems, cutting error-induced
  overhead by roughly 30% and enhancing runtime performance by about 15% for
  certain applications with moderate communication costs.
paper_authors._id: 66666383113312a4a955f263
paper_authors.name: Jianbo Dong
paper_authors._id: 66666383113312a4a955f264
paper_authors.name: Bin Luo
paper_authors._id: 66666383113312a4a955f265
paper_authors.name: Jun Zhang
paper_authors._id: 66666383113312a4a955f266
paper_authors.name: Pengcheng Zhang
paper_authors._id: 66666383113312a4a955f267
paper_authors.name: Fei Feng
paper_authors._id: 66666383113312a4a955f268
paper_authors.name: Yikai Zhu
paper_authors._id: 66666383113312a4a955f269
paper_authors.name: Ang Liu
paper_authors._id: 66666383113312a4a955f26a
paper_authors.name: Zian Chen
paper_authors._id: 66666383113312a4a955f26b
paper_authors.name: Yi Shi
paper_authors._id: 66666383113312a4a955f26c
paper_authors.name: Hairong Jiao
paper_authors._id: 66666383113312a4a955f26d
paper_authors.name: Gang Lu
paper_authors._id: 66666383113312a4a955f26e
paper_authors.name: Yu Guan
paper_authors._id: 66666383113312a4a955f26f
paper_authors.name: Ennan Zhai
paper_authors._id: 66666383113312a4a955f270
paper_authors.name: Wencong Xiao
paper_authors._id: 66666383113312a4a955f271
paper_authors.name: Hanyu Zhao
paper_authors._id: 66666383113312a4a955f272
paper_authors.name: Man Yuan
paper_authors._id: 66666383113312a4a955f273
paper_authors.name: Siran Yang
paper_authors._id: 66666383113312a4a955f274
paper_authors.name: Xiang Li
paper_authors._id: 66666383113312a4a955f275
paper_authors.name: Jiamang Wang
paper_authors._id: 66666383113312a4a955f276
paper_authors.name: Rui Men
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-23T16:30:02.460Z
paper_authors._id: 6209bb200436d7d6f27cbeea
paper_authors.avatarUrl: /avatars/0b8a72a8b66ef7b36780fe2ccc343f78.svg
paper_authors.fullname: Iurnem
paper_authors.type: user
paper_authors.user: Iurnem
paper_authors._id: 66666383113312a4a955f277
paper_authors.name: Jianwei Zhang
paper_authors._id: 66666383113312a4a955f278
paper_authors.name: Huang Zhong
paper_authors._id: 66666383113312a4a955f279
paper_authors.name: Dennis Cai
paper_authors._id: 66666383113312a4a955f27a
paper_authors.name: Yuan Xie
paper_authors._id: 66666383113312a4a955f27b
paper_authors.name: Binzhang Fu
paper_title: Boosting Large-scale Parallel Training Efficiency with C4: A
    Communication-Driven Approach
paper_summary: The emergence of Large Language Models (LLMs) has necessitated the adoption
  of parallel training techniques, involving the deployment of thousands of GPUs
  to train a single model. Unfortunately, we have found that the efficiency of
  current parallel training is often suboptimal, largely due to the following two
  main issues. Firstly, hardware failures are inevitable, leading to
  interruptions in the training tasks. The inability to quickly identify the
  faulty components results in a substantial waste of GPU resources. Secondly,
  since GPUs must wait for parameter synchronization to complete before
  proceeding to the next round of computation, network congestions can greatly
  increase the waiting time for GPUs. To address these challenges, this paper
  introduces a communication-driven solution, namely the C4. The key insights of
  C4 are two folds. First, in parallel training, collective communication
  exhibits periodic and homogeneous characteristics, so any anomalies are
  certainly due to some form of hardware malfunction. By leveraging this feature,
  C4 can rapidly identify the faulty components, swiftly isolate the anomaly, and
  restart the task, thereby avoiding resource wastage caused by delays in anomaly
  detection. Second, the predictable communication model of collective
  communication, involving few large flows, allows C4 to efficiently execute
  traffic planning, substantially reducing network congestion. C4 has been
  extensively implemented across our production systems, cutting error-induced
  overhead by roughly 30% and enhancing runtime performance by about 15% for
  certain applications with moderate communication costs.
paper_ai_keywords: Large Language Models (LLMs) parallel training techniques GPU resources hardware failures parameter synchronization network congestions communication-driven solution C4 collective communication anomaly detection traffic planning runtime performance