title: DeepFlow: Serverless Large Language Model Serving at Scale
summary: This paper introduces DeepFlow, a scalable and serverless AI platform
  designed to efficiently serve large language models (LLMs) at scale in cloud
  environments. DeepFlow addresses key challenges such as resource allocation,
  serving efficiency, and cold start latencies through four main design
  components. First, it uses a simple serverless abstraction called the
  request-job-task model, which helps manage AI workloads across post-training
  and model serving tasks. Second, it builds an in-house serving engine FlowServe
  using a microkernel-inspired design, NPU-centric execution, and SPMD-based
  parallelism to optimize LLM serving. The system also includes novel scheduling
  policies tailored for both PD-disaggregated and PD-colocated configurations.
  With optimizations like pre-warmed pods, DRAM pre-loading, and NPU-fork,
  DeepFlow can scale up to 64 instances in seconds. DeepFlow has been in
  production for over a year, operating on a large Ascend NPU cluster and
  providing industrystandard APIs for fine-tuning, agent serving, and model
  serving to our customers.
paper_authors._id: 679ae0db7b24dd74c70f243e
paper_authors.name: Junhao Hu
paper_authors._id: 679ae0db7b24dd74c70f243f
paper_authors.name: Jiang Xu
paper_authors._id: 679ae0db7b24dd74c70f2440
paper_authors.name: Zhixia Liu
paper_authors._id: 679ae0db7b24dd74c70f2441
paper_authors.name: Yulong He
paper_authors._id: 679ae0db7b24dd74c70f2442
paper_authors.name: Yuetao Chen
paper_authors._id: 679ae0db7b24dd74c70f2443
paper_authors.name: Hao Xu
paper_authors._id: 679ae0db7b24dd74c70f2444
paper_authors.name: Jiang Liu
paper_authors._id: 679ae0db7b24dd74c70f2445
paper_authors.name: Baoquan Zhang
paper_authors._id: 679ae0db7b24dd74c70f2446
paper_authors.name: Shining Wan
paper_authors._id: 679ae0db7b24dd74c70f2447
paper_authors.name: Gengyuan Dan
paper_authors._id: 679ae0db7b24dd74c70f2448
paper_authors.name: Zhiyu Dong
paper_authors._id: 679ae0db7b24dd74c70f2449
paper_authors.name: Zhihao Ren
paper_authors._id: 679ae0db7b24dd74c70f244a
paper_authors.name: Jie Meng
paper_authors._id: 679ae0db7b24dd74c70f244b
paper_authors.name: Chao He
paper_authors._id: 679ae0db7b24dd74c70f244c
paper_authors.name: Changhong Liu
paper_authors._id: 679ae0db7b24dd74c70f244d
paper_authors.name: Tao Xie
paper_authors._id: 679ae0db7b24dd74c70f244e
paper_authors.name: Dayun Lin
paper_authors._id: 679ae0db7b24dd74c70f244f
paper_authors.name: Qin Zhang
paper_authors._id: 679ae0db7b24dd74c70f2450
paper_authors.name: Yue Yu
paper_authors._id: 679ae0db7b24dd74c70f2451
paper_authors.name: Hao Feng
paper_authors._id: 679ae0db7b24dd74c70f2452
paper_authors.name: Xusheng Chen
paper_authors._id: 679ae0db7b24dd74c70f2453
paper_authors.name: Yizhou Shan
paper_title: DeepFlow: Serverless Large Language Model Serving at Scale
paper_summary: This paper introduces DeepFlow, a scalable and serverless AI platform
  designed to efficiently serve large language models (LLMs) at scale in cloud
  environments. DeepFlow addresses key challenges such as resource allocation,
  serving efficiency, and cold start latencies through four main design
  components. First, it uses a simple serverless abstraction called the
  request-job-task model, which helps manage AI workloads across post-training
  and model serving tasks. Second, it builds an in-house serving engine FlowServe
  using a microkernel-inspired design, NPU-centric execution, and SPMD-based
  parallelism to optimize LLM serving. The system also includes novel scheduling
  policies tailored for both PD-disaggregated and PD-colocated configurations.
  With optimizations like pre-warmed pods, DRAM pre-loading, and NPU-fork,
  DeepFlow can scale up to 64 instances in seconds. DeepFlow has been in
  production for over a year, operating on a large Ascend NPU cluster and
  providing industrystandard APIs for fine-tuning, agent serving, and model
  serving to our customers.
paper_ai_keywords: request-job-task model serverless abstraction AI workloads post-training tasks model serving tasks serving engine microkernel-inspired design NPU-centric execution SPMD-based parallelism scheduling policies PD-disaggregated configurations PD-colocated configurations pre-warmed pods DRAM pre-loading NPU-fork Ascend NPU APIs fine-tuning agent serving