ESG: Pipeline-Conscious Efficient Scheduling of DNN Workflows on Serverless Platforms with Shareable GPUs
<|reference_start|>ESG: Pipeline-Conscious Efficient Scheduling of DNN Workflows on Serverless Platforms with Shareable GPUs: Recent years have witnessed increasing interest in machine learning inferences on serverless computing for its auto-scaling and cost effective properties. Existing serverless computing, however, lacks effective job scheduling methods to handle the schedule space dramatically expanded by GPU sharing, task batching, and inter-task relations. Prior solutions have dodged the issue by neglecting some important factors, leaving some large performance potential locked. This paper presents ESG, a new scheduling algorithm that directly addresses the difficulties. ESG treats sharable GPU as a first-order factor in scheduling. It employs an optimality-guided adaptive method by combining A*-search and a novel dual-blade pruning to dramatically prune the scheduling space without compromising the quality. It further introduces a novel method, dominator-based SLO distribution, to ensure the scalability of the scheduler. The results show that ESG can significantly improve the SLO hit rates 61%-80% while saving 47%-187% costs over prior work.<|reference_end|>
@article{hui2024esg:,
    title={ESG: Pipeline-Conscious Efficient Scheduling of DNN Workflows on
  Serverless Platforms with Shareable GPUs},
    author={Xinning Hui, Yuanchao Xu, Zhishan Guo and Xipeng Shen},
    journal={arXiv preprint arXiv:2404.16812},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.16812},
    primaryClass={cs.DC}
}