Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services
<|reference_start|>Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services: The advent of large language models (LLMs) has transformed text-based services, enabling capabilities ranging from real-time translation to AI-driven chatbots. However, existing serving systems primarily focus on optimizing server-side aggregate metrics like token generation throughput, ignoring individual user experience with streamed text. As a result, under high and/or bursty load, a significant number of users can receive unfavorable service quality or poor Quality-of-Experience (QoE). In this paper, we first formally define QoE of text streaming services, where text is delivered incrementally and interactively to users, by considering the end-to-end token delivery process throughout the entire interaction with the user. Thereafter, we propose Andes, a QoE-aware serving system that enhances user experience for LLM-enabled text streaming services. At its core, Andes strategically allocates contended GPU resources among multiple requests over time to optimize their QoE. Our evaluations demonstrate that, compared to the state-of-the-art LLM serving systems like vLLM, Andes improves the average QoE by up to 3.2$\times$ under high request rate, or alternatively, it attains up to 1.6$\times$ higher request rate while preserving high QoE.<|reference_end|>
@article{liu2024andes:,
    title={Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text
  Streaming Services},
    author={Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, Mosharaf
  Chowdhury},
    journal={arXiv preprint arXiv:2404.16283},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.16283},
    primaryClass={cs.DC cs.LG}
}