Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling
<|reference_start|>Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling: Mobile and IoT applications increasingly adopt deep learning inference to provide intelligence. Inference requests are typically sent to a cloud infrastructure over a wireless network that is highly variable, leading to the challenge of dynamic Service Level Objectives (SLOs) at the request level. This paper presents Sponge, a novel deep learning inference serving system that maximizes resource efficiency while guaranteeing dynamic SLOs. Sponge achieves its goal by applying in-place vertical scaling, dynamic batching, and request reordering. Specifically, we introduce an Integer Programming formulation to capture the resource allocation problem, providing a mathematical model of the relationship between latency, batch size, and resources. We demonstrate the potential of Sponge through a prototype implementation and preliminary experiments and discuss future works.<|reference_end|>
@article{razavi2024sponge:,
    title={Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical
  Scaling},
    author={Kamran Razavi, Saeid Ghafouri, Max M"uhlh"auser, Pooyan Jamshidi,
  Lin Wang},
    journal={arXiv preprint arXiv:2404.00704},
    year={2024},
    doi={10.1145/3642970.3655833},
    archivePrefix={arXiv},
    eprint={2404.00704},
    primaryClass={cs.DC}
}