ForestColl: Efficient Collective Communications on Heterogeneous Network Fabrics
<|reference_start|>ForestColl: Efficient Collective Communications on Heterogeneous Network Fabrics: As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient communication schedules is challenging, given today's highly diverse and heterogeneous network fabrics. In this paper, we present ForestColl, a tool that generates performant schedules for any network topology. ForestColl constructs broadcast/aggregation spanning trees as the communication schedule, achieving theoretically optimal throughput. Its schedule generation runs in strongly polynomial time and is highly scalable. ForestColl supports any network fabric, including both switching fabrics and direct connections. We evaluated ForestColl on multi-box AMD MI250 and NVIDIA DGX A100 platforms. ForestColl's schedules delivered up to 130% higher performance compared to the vendors' own optimized communication libraries, RCCL and NCCL, and achieved a 20% speedup in LLM training. ForestColl also outperforms other state-of-the-art schedule generation techniques with both up to 61% more efficient generated schedules and orders of magnitude faster schedule generation speed.<|reference_end|>
@article{zhao2024forestcoll:,
    title={ForestColl: Efficient Collective Communications on Heterogeneous Network
  Fabrics},
    author={Liangyu Zhao and Saeed Maleki and Aashaka Shah and Ziyue Yang and
  Hossein Pourreza and Arvind Krishnamurthy},
    journal={arXiv preprint arXiv:2402.06787},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.06787},
    primaryClass={cs.NI cs.DC cs.LG}
}