HiCCL: A Hierarchical Collective Communication Library
<|reference_start|>HiCCL: A Hierarchical Collective Communication Library: HiCCL (Hierarchical Collective Communication Library) addresses the growing complexity and diversity in high-performance network architectures. As GPU systems have envolved into networks of GPUs with different multilevel communication hierarchies, optimizing each collective function for a specific system has become a challenging task. Consequently, many collective libraries struggle to adapt to different hardware and software, especially across systems from different vendors. HiCCL's library design decouples the collective communication logic from network-specific optimizations through a compositional API. The communication logic is composed using multicast, reduction, and fence primitives, which are then factorized for a specified network hieararchy using only point-to-point operations within a level. Finally, striping and pipelining optimizations applied as specified for streamlining the execution.   Performance evaluation of HiCCL across four different machines$\unicode{x2014}$two with Nvidia GPUs, one with AMD GPUs, and one with Intel GPUs$\unicode{x2014}$demonstrates an average 17$\times$ higher throughput than the collectives of highly specialized GPU-aware MPI implementations, and competitive throughput with those of vendor-specific libraries (NCCL, RCCL, and OneCCL), while providing portability across all four machines.<|reference_end|>
@article{hidayetoglu2024hiccl:,
    title={HiCCL: A Hierarchical Collective Communication Library},
    author={Mert Hidayetoglu, Simon Garcia de Gonzalo, Elliott Slaughter, Pinku
  Surana, Wen-mei Hwu, William Gropp, Alex Aiken},
    journal={arXiv preprint arXiv:2408.05962},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.05962},
    primaryClass={cs.DC}
}