Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
<|reference_start|>Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning: The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC.<|reference_end|>
@article{an2024fire-flyer,
    title={Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep
  Learning},
    author={Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui
  Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang
  Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li, Wenfeng Liang, Xiaodong Liu,
  Xin Liu, Yiyuan Liu, Yuxuan Liu, Shanghao Lu, Xuan Lu, Xiaotao Nie, Tian Pei,
  Junjie Qiu, Hui Qu, Zehui Ren, Zhangli Sha, Xuecheng Su, Xiaowen Sun, Yixuan
  Tan, Minghui Tang, Shiyu Wang, Yaohui Wang, Yongji Wang, Ziwei Xie, Yiliang
  Xiong, Yanhong Xu, Shengfeng Ye, Shuiping Yu, Yukun Zha, Liyue Zhang, Haowei
  Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao,
  Shangyan Zhou, Shunfeng Zhou, Yuheng Zou},
    journal={arXiv preprint arXiv:2408.14158},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.14158},
    primaryClass={cs.DC cs.AI}
}