Towards Pareto Optimal Throughput in Small Language Model Serving
<|reference_start|>Towards Pareto Optimal Throughput in Small Language Model Serving: Large language models (LLMs) have revolutionized the state-of-the-art of many different natural language processing tasks. Although serving LLMs is computationally and memory demanding, the rise of Small Language Models (SLMs) offers new opportunities for resource-constrained users, who now are able to serve small models with cutting-edge performance. In this paper, we present a set of experiments designed to benchmark SLM inference at performance and energy levels. Our analysis provides a new perspective in serving, highlighting that the small memory footprint of SLMs allows for reaching the Pareto-optimal throughput within the resource capacity of a single accelerator. In this regard, we present an initial set of findings demonstrating how model replication can effectively improve resource utilization for serving SLMs.<|reference_end|>
@article{recasens2024towards,
    title={Towards Pareto Optimal Throughput in Small Language Model Serving},
    author={Pol G.Recasens and Yue Zhu and Chen Wang and Eun Kyung Lee and Olivier
  Tardieu and Alaa Youssef and Jordi Torres and Josep Ll. Berral},
    journal={arXiv preprint arXiv:2404.03353},
    year={2024},
    doi={10.1145/3642970.3655832},
    archivePrefix={arXiv},
    eprint={2404.03353},
    primaryClass={cs.CL}
}