title: S-LoRA: Serving Thousands of Concurrent LoRA Adapters
summary: The "pretrain-then-finetune" paradigm is commonly adopted in the deployment
  of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient
  fine-tuning method, is often employed to adapt a base model to a multitude of
  tasks, resulting in a substantial collection of LoRA adapters derived from one
  base model. We observe that this paradigm presents significant opportunities
  for batched inference during serving. To capitalize on these opportunities, we
  present S-LoRA, a system designed for the scalable serving of many LoRA
  adapters. S-LoRA stores all adapters in the main memory and fetches the
  adapters used by the currently running queries to the GPU memory. To
  efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes
  Unified Paging. Unified Paging uses a unified memory pool to manage dynamic
  adapter weights with different ranks and KV cache tensors with varying sequence
  lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and
  highly optimized custom CUDA kernels for heterogeneous batching of LoRA
  computation. Collectively, these features enable S-LoRA to serve thousands of
  LoRA adapters on a single GPU or across multiple GPUs with a small overhead.
  Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with
  naive support of LoRA serving), S-LoRA can improve the throughput by up to 4
  times and increase the number of served adapters by several orders of
  magnitude. As a result, S-LoRA enables scalable serving of many task-specific
  fine-tuned models and offers the potential for large-scale customized
  fine-tuning services.
paper_authors._id: 6549c8e17ab20d807df276dd
paper_authors.name: Ying Sheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:45:44.239Z
paper_authors._id: 6455b969654d8bccae50736c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6455b969654d8bccae50736c/WsK0_sbQSVeosi-fh-O_U.jpeg
paper_authors.fullname: Ying Sheng
paper_authors.type: user
paper_authors.user: ying1123
paper_authors._id: 6549c8e17ab20d807df276de
paper_authors.name: Shiyi Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:45:51.245Z
paper_authors._id: 64ebbae6895a36ab28de811a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ebbae6895a36ab28de811a/gBiaQP4paS4L13eu-yRm7.jpeg
paper_authors.fullname: Shiyi Cao
paper_authors.type: user
paper_authors.user: eva98
paper_authors._id: 6549c8e17ab20d807df276df
paper_authors.name: Dacheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:45:59.515Z
paper_authors._id: 63715b25ffc0489ed7d1f415
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63715b25ffc0489ed7d1f415/xZJepbs0LRqFbW1knnBKR.jpeg
paper_authors.fullname: Dacheng Li
paper_authors.type: user
paper_authors.user: DachengLi
paper_authors._id: 6549c8e17ab20d807df276e0
paper_authors.name: Coleman Hooper
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:46:06.278Z
paper_authors._id: 647d650483cdb93baf77dd8f
paper_authors.avatarUrl: /avatars/03a076088efb009a5fe5f27033569320.svg
paper_authors.fullname: Coleman Hooper
paper_authors.type: user
paper_authors.user: chooper1
paper_authors._id: 6549c8e17ab20d807df276e1
paper_authors.name: Nicholas Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:02:30.645Z
paper_authors._id: 656a4547996819a82836852d
paper_authors.avatarUrl: /avatars/a5b522c18fced97140a156ff3e88d2e8.svg
paper_authors.fullname: Nicholas Lee
paper_authors.type: user
paper_authors.user: nicholaslee
paper_authors._id: 6549c8e17ab20d807df276e2
paper_authors.name: Shuo Yang
paper_authors._id: 6549c8e17ab20d807df276e3
paper_authors.name: Christopher Chou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:47:13.582Z
paper_authors._id: 62fb0555e8c9c532aa7c6e5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fb0555e8c9c532aa7c6e5d/AUds_M-PA7TCrj3gQ15IN.jpeg
paper_authors.fullname: Christopher Chou
paper_authors.type: user
paper_authors.user: BabyChou
paper_authors._id: 6549c8e17ab20d807df276e4
paper_authors.name: Banghua Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:47:24.274Z
paper_authors._id: 647b8885aba7062fe5c32000
paper_authors.avatarUrl: /avatars/128ceae78490110ae41202851e84d58e.svg
paper_authors.fullname: Banghua Zhu
paper_authors.type: user
paper_authors.user: banghua
paper_authors._id: 6549c8e17ab20d807df276e5
paper_authors.name: Lianmin Zheng
paper_authors._id: 6549c8e17ab20d807df276e6
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:47:49.327Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 6549c8e17ab20d807df276e7
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:48:03.399Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 6549c8e17ab20d807df276e8
paper_authors.name: Ion Stoica
paper_title: S-LoRA: Serving Thousands of Concurrent LoRA Adapters
paper_summary: The "pretrain-then-finetune" paradigm is commonly adopted in the deployment
  of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient
  fine-tuning method, is often employed to adapt a base model to a multitude of
  tasks, resulting in a substantial collection of LoRA adapters derived from one
  base model. We observe that this paradigm presents significant opportunities
  for batched inference during serving. To capitalize on these opportunities, we
  present S-LoRA, a system designed for the scalable serving of many LoRA
  adapters. S-LoRA stores all adapters in the main memory and fetches the
  adapters used by the currently running queries to the GPU memory. To
  efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes
  Unified Paging. Unified Paging uses a unified memory pool to manage dynamic
  adapter weights with different ranks and KV cache tensors with varying sequence
  lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and
  highly optimized custom CUDA kernels for heterogeneous batching of LoRA
  computation. Collectively, these features enable S-LoRA to serve thousands of
  LoRA adapters on a single GPU or across multiple GPUs with a small overhead.
  Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with
  naive support of LoRA serving), S-LoRA can improve the throughput by up to 4
  times and increase the number of served adapters by several orders of
  magnitude. As a result, S-LoRA enables scalable serving of many task-specific
  fine-tuned models and offers the potential for large-scale customized
  fine-tuning services.
paper_ai_keywords: Low-Rank Adaptation (LoRA) parameter-efficient fine-tuning batched inference S-LoRA Unified Paging tensor parallelism custom CUDA kernels heterogeneous batching KV cache tensors sequence lengths HuggingFace PEFT vLLM