Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems
<|reference_start|>Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems: The rapid adoption of large language models (LLMs) has led to significant advances in natural language processing and text generation. However, the energy consumed through LLM model inference remains a major challenge for sustainable AI deployment. To address this problem, we model the workload-dependent energy consumption and runtime of LLM inference tasks on heterogeneous GPU-CPU systems. By conducting an extensive characterization study of several state-of-the-art LLMs and analyzing their energy and runtime behavior across different magnitudes of input prompts and output text, we develop accurate (R^2>0.96) energy and runtime models for each LLM. We employ these models to explore an offline, energy-optimal LLM workload scheduling framework. Through a case study, we demonstrate the advantages of energy and accuracy aware scheduling compared to existing best practices.<|reference_end|>
@article{wilkins2024offline,
    title={Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM
  Inference on Heterogeneous Systems},
    author={Grant Wilkins and Srinivasan Keshav and Richard Mortier},
    journal={arXiv preprint arXiv:2407.04014},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.04014},
    primaryClass={cs.DC}
}