title: PowerInfer-2: Fast Large Language Model Inference on a Smartphone
summary: This paper introduces PowerInfer-2, a framework designed for high-speed
  inference of Large Language Models (LLMs) on smartphones, particularly
  effective for models whose sizes exceed the device's memory capacity. The key
  insight of PowerInfer-2 is to utilize the heterogeneous computation, memory,
  and I/O resources in smartphones by decomposing traditional matrix computations
  into fine-grained neuron cluster computations. Specifically, PowerInfer-2
  features a polymorphic neuron engine that adapts computational strategies for
  various stages of LLM inference. Additionally, it introduces segmented neuron
  caching and fine-grained neuron-cluster-level pipelining, which effectively
  minimize and conceal the overhead caused by I/O operations. The implementation
  and evaluation of PowerInfer-2 demonstrate its capability to support a wide
  array of LLM models on two smartphones, achieving up to a 29.2x speed increase
  compared with state-of-the-art frameworks. Notably, PowerInfer-2 is the first
  system to serve the TurboSparse-Mixtral-47B model with a generation rate of
  11.68 tokens per second on a smartphone. For models that fit entirely within
  the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory
  usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM.
  For more details, including a demonstration video, please visit the project
  site at www.powerinfer.ai/v2.
paper_authors._id: 666a76d52c3271b9b5cb02b7
paper_authors.name: Zhenliang Xue
paper_authors._id: 666a76d52c3271b9b5cb02b8
paper_authors.name: Yixin Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T12:02:16.105Z
paper_authors._id: 642924f916d4d8293c93af08
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642924f916d4d8293c93af08/Olp4zFPgGu2E6mFFPXfQO.jpeg
paper_authors.fullname: Yixin Song
paper_authors.type: user
paper_authors.user: yixinsong
paper_authors._id: 666a76d52c3271b9b5cb02b9
paper_authors.name: Zeyu Mi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:30:30.510Z
paper_authors._id: 6565a74575968d50cc1e51d3
paper_authors.avatarUrl: /avatars/cc50e33b4e275aff2c017b7a2d7830e4.svg
paper_authors.fullname: Zeyu Mi
paper_authors.type: user
paper_authors.user: yzmizeyu
paper_authors._id: 666a76d52c3271b9b5cb02ba
paper_authors.name: Le Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T12:02:24.450Z
paper_authors._id: 651ac281a9a91bf39df6065d
paper_authors.avatarUrl: /avatars/6fbf46f3ca32d1a63137255b93b534b4.svg
paper_authors.fullname: LeChen
paper_authors.type: user
paper_authors.user: LeChen
paper_authors._id: 666a76d52c3271b9b5cb02bb
paper_authors.name: Yubin Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T12:02:29.842Z
paper_authors._id: 6403257d56038547951d77c9
paper_authors.avatarUrl: /avatars/6a7e0ac7229cd48c603e2d1a1677707b.svg
paper_authors.fullname: Yubin Xia
paper_authors.type: user
paper_authors.user: unitial
paper_authors._id: 666a76d52c3271b9b5cb02bc
paper_authors.name: Haibo Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T12:02:43.910Z
paper_authors._id: 661f4119cc26dfa68e2e1f54
paper_authors.avatarUrl: /avatars/351936ebb0b015d152645844a9d094b3.svg
paper_authors.fullname: chenhaibo
paper_authors.type: user
paper_authors.user: hbchen07
paper_title: PowerInfer-2: Fast Large Language Model Inference on a Smartphone
paper_summary: This paper introduces PowerInfer-2, a framework designed for high-speed
  inference of Large Language Models (LLMs) on smartphones, particularly
  effective for models whose sizes exceed the device's memory capacity. The key
  insight of PowerInfer-2 is to utilize the heterogeneous computation, memory,
  and I/O resources in smartphones by decomposing traditional matrix computations
  into fine-grained neuron cluster computations. Specifically, PowerInfer-2
  features a polymorphic neuron engine that adapts computational strategies for
  various stages of LLM inference. Additionally, it introduces segmented neuron
  caching and fine-grained neuron-cluster-level pipelining, which effectively
  minimize and conceal the overhead caused by I/O operations. The implementation
  and evaluation of PowerInfer-2 demonstrate its capability to support a wide
  array of LLM models on two smartphones, achieving up to a 29.2x speed increase
  compared with state-of-the-art frameworks. Notably, PowerInfer-2 is the first
  system to serve the TurboSparse-Mixtral-47B model with a generation rate of
  11.68 tokens per second on a smartphone. For models that fit entirely within
  the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory
  usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM.
  For more details, including a demonstration video, please visit the project
  site at www.powerinfer.ai/v2.
paper_ai_keywords: polymorphic neuron engine neuron cluster computations segmented neuron caching neuron-cluster-level pipelining TurboSparse-Mixtral-47B llama.cpp MLC-LLM