D\'ej\`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving
<|reference_start|>D\'ej\`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving: Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. In this paper, we propose D\'ej\`aVu, a system to address all these challenges using a versatile and efficient KV cache streaming library (D\'ej\`aVuLib). Using D\'ej\`aVuLib, we propose and implement efficient prompt-token disaggregation to reduce pipeline bubbles, microbatch swapping for efficient GPU memory management, and state replication for fault-tolerance. We highlight the efficacy of these solutions on a range of large models across cloud deployments.<|reference_end|>
@article{strati2024d\'ej\`avu:,
    title={D\'ej\`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM
  Serving},
    author={Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski,
  Ana Klimovic},
    journal={arXiv preprint arXiv:2403.01876},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.01876},
    primaryClass={cs.DC}
}