A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
<|reference_start|>A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models: The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.<|reference_end|>
@article{guo2024a,
    title={A Survey: Collaborative Hardware and Software Design in the Era of Large
  Language Models},
    author={Cong Guo, Feng Cheng, Zhixu Du, James Kiessling, Jonathan Ku, Shiyu
  Li, Ziru Li, Mingyuan Ma, Tergel Molom-Ochir, Benjamin Morris, Haoxuan Shan,
  Jingwei Sun, Yitu Wang, Chiyue Wei, Xueying Wu, Yuhao Wu, Hao Frank Yang,
  Jingyang Zhang, Junyao Zhang, Qilin Zheng, Guanglei Zhou, Hai (Helen) Li,
  Yiran Chen},
    journal={arXiv preprint arXiv:2410.07265},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07265},
    primaryClass={cs.AR cs.AI cs.LG cs.SE}
}