title: MemServe: Context Caching for Disaggregated LLM Serving with Elastic
    Memory Pool
summary: Large language model (LLM) serving has transformed from stateless to stateful
  systems, utilizing techniques like context caching and disaggregated inference.
  These optimizations extend the lifespan and domain of the KV cache,
  necessitating a new architectural approach. We present MemServe, a unified
  system that integrates both inter-request and intra-request optimizations.
  MemServe introduces MemPool, an elastic memory pool managing distributed memory
  and KV caches across serving instances. Using MemPool APIs, MemServe combines
  context caching with disaggregated inference for the first time, supported by a
  global scheduler that enhances cache reuse through a global prompt tree-based
  locality-aware policy. Tests show that MemServe significantly improves job
  completion time and time-to-first-time.
paper_authors._id: 667cab9120d68cc58172b03d
paper_authors.name: Cunchen Hu
paper_authors._id: 667cab9120d68cc58172b03e
paper_authors.name: Heyang Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:55:21.029Z
paper_authors._id: 647fbf98095af0bf116d4de6
paper_authors.avatarUrl: /avatars/c449c9a13898590da47faa378b41e20c.svg
paper_authors.fullname: Heyang Huang
paper_authors.type: user
paper_authors.user: heyangh
paper_authors._id: 667cab9120d68cc58172b03f
paper_authors.name: Junhao Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-27T18:55:28.586Z
paper_authors._id: 655336cb0166ff6bd23a979f
paper_authors.avatarUrl: /avatars/6b0d8a8064dbcf4db895ca6735dfa0a2.svg
paper_authors.fullname: Hu
paper_authors.type: user
paper_authors.user: Junhaohu
paper_authors._id: 667cab9120d68cc58172b040
paper_authors.name: Jiang Xu
paper_authors._id: 667cab9120d68cc58172b041
paper_authors.name: Xusheng Chen
paper_authors._id: 667cab9120d68cc58172b042
paper_authors.name: Tao Xie
paper_authors._id: 667cab9120d68cc58172b043
paper_authors.name: Chenxi Wang
paper_authors._id: 667cab9120d68cc58172b044
paper_authors.name: Sa Wang
paper_authors._id: 667cab9120d68cc58172b045
paper_authors.name: Yungang Bao
paper_authors._id: 667cab9120d68cc58172b046
paper_authors.name: Ninghui Sun
paper_authors._id: 667cab9120d68cc58172b047
paper_authors.name: Yizhou Shan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:02:32.242Z
paper_authors._id: 6457885a75f8f7d26aa5bc44
paper_authors.avatarUrl: /avatars/8ce57c4d60a1f1b5afa2c592207a8335.svg
paper_authors.fullname: allthingsdisaggregated
paper_authors.type: user
paper_authors.user: lastweek
paper_title: MemServe: Context Caching for Disaggregated LLM Serving with Elastic
    Memory Pool
paper_summary: Large language model (LLM) serving has transformed from stateless to stateful
  systems, utilizing techniques like context caching and disaggregated inference.
  These optimizations extend the lifespan and domain of the KV cache,
  necessitating a new architectural approach. We present MemServe, a unified
  system that integrates both inter-request and intra-request optimizations.
  MemServe introduces MemPool, an elastic memory pool managing distributed memory
  and KV caches across serving instances. Using MemPool APIs, MemServe combines
  context caching with disaggregated inference for the first time, supported by a
  global scheduler that enhances cache reuse through a global prompt tree-based
  locality-aware policy. Tests show that MemServe significantly improves job
  completion time and time-to-first-time.
paper_ai_keywords: Neural Context caching Disaggregated inference MemPool Elastic memory pool KV caches Scheduler Prompt tree-based LocM Locality-aware policy OOD generalization