title: T3: Transparent Tracking & Triggering for Fine-grained Overlap of
    Compute & Collectives
summary: Large Language Models increasingly rely on distributed techniques for their
  training and inference. These techniques require communication across devices
  which can reduce scaling efficiency as the number of devices increases. While
  some distributed techniques can overlap, and thus, hide this communication with
  independent computations, techniques such as Tensor Parallelism (TP) inherently
  serialize communication with model execution. One approach to hide this
  serialized communication is to interleave it with the producer operation (of
  the communicated data) in a fine-grained manner. However, this fine-grained
  interleaving of communication and computation in software can be difficult.
  Furthermore, as with any concurrent execution, it requires compute and memory
  resources to be shared between computation and communication, causing resource
  contention that reduces overlapping efficacy.
    To overcome these challenges, we propose T3 which applies hardware-software
  co-design to transparently overlap serialized communication while minimizing
  resource contention with compute. T3 transparently fuses producer operations
  with the subsequent communication via a simple configuration of the producer's
  output address space and requires minor software changes. At the hardware
  level, T3 adds a lightweight track and trigger mechanism to orchestrate the
  producer's compute, and communication. It further uses compute-enhanced
  memories for communication's attendant compute. As a result, T3 reduces
  resource contention, and efficiently overlaps serialized communication with
  computation. For important Transformer models like T-NLG, T3 speeds up
  communication-heavy sublayers by 30% geomean (max 47%) and reduces data
  movement by 22% geomean (max 36%). Furthermore, T3's benefits persist as models
  scale: geomean 29% for sublayers in sim500-billion parameter models, PALM
  and MT-NLG.
paper_authors._id: 65b9b31d1f5a8e1a0eb476e6
paper_authors.name: Suchita Pati
paper_authors._id: 65b9b31d1f5a8e1a0eb476e7
paper_authors.name: Shaizeen Aga
paper_authors._id: 65b9b31d1f5a8e1a0eb476e8
paper_authors.name: Mahzabeen Islam
paper_authors._id: 65b9b31d1f5a8e1a0eb476e9
paper_authors.name: Nuwan Jayasena
paper_authors._id: 65b9b31d1f5a8e1a0eb476ea
paper_authors.name: Matthew D. Sinclair
paper_title: T3: Transparent Tracking & Triggering for Fine-grained Overlap of
    Compute & Collectives
paper_summary: Large Language Models increasingly rely on distributed techniques for their
  training and inference. These techniques require communication across devices
  which can reduce scaling efficiency as the number of devices increases. While
  some distributed techniques can overlap, and thus, hide this communication with
  independent computations, techniques such as Tensor Parallelism (TP) inherently
  serialize communication with model execution. One approach to hide this
  serialized communication is to interleave it with the producer operation (of
  the communicated data) in a fine-grained manner. However, this fine-grained
  interleaving of communication and computation in software can be difficult.
  Furthermore, as with any concurrent execution, it requires compute and memory
  resources to be shared between computation and communication, causing resource
  contention that reduces overlapping efficacy.
    To overcome these challenges, we propose T3 which applies hardware-software
  co-design to transparently overlap serialized communication while minimizing
  resource contention with compute. T3 transparently fuses producer operations
  with the subsequent communication via a simple configuration of the producer's
  output address space and requires minor software changes. At the hardware
  level, T3 adds a lightweight track and trigger mechanism to orchestrate the
  producer's compute, and communication. It further uses compute-enhanced
  memories for communication's attendant compute. As a result, T3 reduces
  resource contention, and efficiently overlaps serialized communication with
  computation. For important Transformer models like T-NLG, T3 speeds up
  communication-heavy sublayers by 30% geomean (max 47%) and reduces data
  movement by 22% geomean (max 36%). Furthermore, T3's benefits persist as models
  scale: geomean 29% for sublayers in sim500-billion parameter models, PALM
  and MT-NLG.
paper_ai_summary: T3 employs hardware-software co-design to optimize the overlap between serialized communication and computation in distributed large language models, improving efficiency and reducing resource contention.
paper_ai_keywords: Tensor Parallelism T3 producer operations communication compute resource contention lightweight track and trigger mechanism compute-enhanced memories Transformer models T-NLG PALM MT-NLG