title: NanoFlow: Towards Optimal Large Language Model Serving Throughput
summary: The increasing usage of Large Language Models (LLMs) has resulted in a
  surging demand for planet-scale serving systems, where tens of thousands of
  GPUs continuously serve hundreds of millions of users. Consequently, throughput
  (under reasonable latency constraints) has emerged as a key metric that
  determines serving systems' performance. To boost throughput, various methods
  of inter-device parallelism (e.g., data, tensor, pipeline) have been explored.
  However, existing methods do not consider overlapping the utilization of
  different resources within a single device, leading to underutilization and
  sub-optimal performance.
    We propose NanoFlow, a novel serving framework that exploits intra-device
  parallelism, which overlaps the usage of resources including compute, memory,
  and network within a single device through operation co-scheduling. To exploit
  intra-device parallelism, NanoFlow introduces two key innovations: First,
  NanoFlow splits requests into nano-batches at the granularity of operations,
  which breaks the dependency of sequential operations in LLM inference and
  enables overlapping; then, to get benefit from overlapping, NanoFlow uses an
  operation-level pipeline with execution unit scheduling, which partitions the
  device's functional units and simultaneously executes different operations in
  each unit. NanoFlow automates the pipeline setup using a parameter search
  algorithm, which enables easily porting NanoFlow to different models. We
  implement NanoFlow on NVIDIA GPUs and evaluate end-to-end serving throughput on
  several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc..
  With practical workloads, NanoFlow provides 1.91x throughput boost compared to
  state-of-the-art serving systems achieving 59% to 72% of optimal throughput
  across ported models.
paper_authors._id: 66cd91e1db7d0dd313163bd6
paper_authors.name: Kan Zhu
paper_authors._id: 66cd91e1db7d0dd313163bd7
paper_authors.name: Yilong Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T20:42:15.634Z
paper_authors._id: 6549b0a808775ce78e535c6a
paper_authors.avatarUrl: /avatars/942066356843d0c424375937f157c975.svg
paper_authors.fullname: Yilong Zhao
paper_authors.type: user
paper_authors.user: ylzhao
paper_authors._id: 66cd91e1db7d0dd313163bd8
paper_authors.name: Liangyu Zhao
paper_authors._id: 66cd91e1db7d0dd313163bd9
paper_authors.name: Gefei Zuo
paper_authors._id: 66cd91e1db7d0dd313163bda
paper_authors.name: Yile Gu
paper_authors._id: 66cd91e1db7d0dd313163bdb
paper_authors.name: Dedong Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T08:00:46.556Z
paper_authors._id: 66ceb78280928507bf033904
paper_authors.avatarUrl: /avatars/3de22015cd2d030d67b700e8d347e7d8.svg
paper_authors.fullname: Dedong Xie
paper_authors.type: user
paper_authors.user: ddxxdd
paper_authors._id: 66cd91e1db7d0dd313163bdc
paper_authors.name: Yufei Gao
paper_authors._id: 66cd91e1db7d0dd313163bdd
paper_authors.name: Qinyu Xu
paper_authors._id: 66cd91e1db7d0dd313163bde
paper_authors.name: Tian Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T08:00:53.913Z
paper_authors._id: 65de79aa3aec0530560cbbc6
paper_authors.avatarUrl: /avatars/fe6db0ee257c5240aa7984f12fd21c79.svg
paper_authors.fullname: Tian Tang
paper_authors.type: user
paper_authors.user: tangt666
paper_authors._id: 66cd91e1db7d0dd313163bdf
paper_authors.name: Zihao Ye
paper_authors._id: 66cd91e1db7d0dd313163be0
paper_authors.name: Keisuke Kamahori
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T08:00:49.051Z
paper_authors._id: 6304ac1a412a1b9d381ca378
paper_authors.avatarUrl: /avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
paper_authors.fullname: Keisuke Kamahori
paper_authors.type: user
paper_authors.user: kamahori
paper_authors._id: 66cd91e1db7d0dd313163be1
paper_authors.name: Chien-Yu Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T20:42:13.730Z
paper_authors._id: 64d1950a0fbfb00b919e9481
paper_authors.avatarUrl: /avatars/7a5f367882012bd08a3ca73f9acd9912.svg
paper_authors.fullname: Chien-Yu Lin
paper_authors.type: user
paper_authors.user: cylinbao
paper_authors._id: 66cd91e1db7d0dd313163be2
paper_authors.name: Stephanie Wang
paper_authors._id: 66cd91e1db7d0dd313163be3
paper_authors.name: Arvind Krishnamurthy
paper_authors._id: 66cd91e1db7d0dd313163be4
paper_authors.name: Baris Kasikci
paper_title: NanoFlow: Towards Optimal Large Language Model Serving Throughput
paper_summary: The increasing usage of Large Language Models (LLMs) has resulted in a
  surging demand for planet-scale serving systems, where tens of thousands of
  GPUs continuously serve hundreds of millions of users. Consequently, throughput
  (under reasonable latency constraints) has emerged as a key metric that
  determines serving systems' performance. To boost throughput, various methods
  of inter-device parallelism (e.g., data, tensor, pipeline) have been explored.
  However, existing methods do not consider overlapping the utilization of
  different resources within a single device, leading to underutilization and
  sub-optimal performance.
    We propose NanoFlow, a novel serving framework that exploits intra-device
  parallelism, which overlaps the usage of resources including compute, memory,
  and network within a single device through operation co-scheduling. To exploit
  intra-device parallelism, NanoFlow introduces two key innovations: First,
  NanoFlow splits requests into nano-batches at the granularity of operations,
  which breaks the dependency of sequential operations in LLM inference and
  enables overlapping; then, to get benefit from overlapping, NanoFlow uses an
  operation-level pipeline with execution unit scheduling, which partitions the
  device's functional units and simultaneously executes different operations in
  each unit. NanoFlow automates the pipeline setup using a parameter search
  algorithm, which enables easily porting NanoFlow to different models. We
  implement NanoFlow on NVIDIA GPUs and evaluate end-to-end serving throughput on
  several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc..
  With practical workloads, NanoFlow provides 1.91x throughput boost compared to
  state-of-the-art serving systems achieving 59% to 72% of optimal throughput
  across ported models.
paper_ai_summary: NanoFlow enhances serving throughput by exploiting intra-device parallelism through operation co-scheduling and execution unit scheduling on GPUs.
paper_ai_keywords: Large Language Models inter-device parallelism intra-device parallelism nano-batches operation co-scheduling operation-level pipeline execution unit scheduling parameter search algorithm