title: Optimized Network Architectures for Large Language Model Training with
    Billions of Parameters
summary: This paper challenges the well-established paradigm for building any-to-any
  networks for training Large Language Models (LLMs). We show that LLMs exhibit a
  unique communication pattern where only small groups of GPUs require
  high-bandwidth any-to-any communication within them, to achieve near-optimal
  training performance. Across these groups of GPUs, the communication is
  insignificant, sparse, and homogeneous. We propose a new network architecture
  that closely resembles the communication requirement of LLMs. Our architecture
  partitions the cluster into sets of GPUs interconnected with non-blocking
  any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
  domains, the network only connects GPUs with communication demands. We call
  this network a "rail-only" connection, and show that our proposed architecture
  reduces the network cost by up to 75% compared to the state-of-the-art
  any-to-any Clos networks without compromising the performance of LLM training.
paper_authors._id: 64bf3752af05eb17c71abf36
paper_authors.name: Weiyang Wang
paper_authors._id: 64bf3752af05eb17c71abf37
paper_authors.name: Manya Ghobadi
paper_authors._id: 64bf3752af05eb17c71abf38
paper_authors.name: Kayvon Shakeri
paper_authors._id: 64bf3752af05eb17c71abf39
paper_authors.name: Ying Zhang
paper_authors._id: 64bf3752af05eb17c71abf3a
paper_authors.name: Naader Hasani
paper_title: Optimized Network Architectures for Large Language Model Training with
    Billions of Parameters
paper_summary: This paper challenges the well-established paradigm for building any-to-any
  networks for training Large Language Models (LLMs). We show that LLMs exhibit a
  unique communication pattern where only small groups of GPUs require
  high-bandwidth any-to-any communication within them, to achieve near-optimal
  training performance. Across these groups of GPUs, the communication is
  insignificant, sparse, and homogeneous. We propose a new network architecture
  that closely resembles the communication requirement of LLMs. Our architecture
  partitions the cluster into sets of GPUs interconnected with non-blocking
  any-to-any high-bandwidth interconnects that we call HB domains. Across the HB
  domains, the network only connects GPUs with communication demands. We call
  this network a "rail-only" connection, and show that our proposed architecture
  reduces the network cost by up to 75% compared to the state-of-the-art
  any-to-any Clos networks without compromising the performance of LLM training.
paper_ai_keywords: any-to-any networks Large Language Models (LLMs) high-bandwidth communication node cases HB domains "rail-only" connection network cost Clos networks image training