title: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
summary: This paper introduces PowerInfer, a high-speed Large Language Model (LLM)
  inference engine on a personal computer (PC) equipped with a single
  consumer-grade GPU. The key underlying the design of PowerInfer is exploiting
  the high locality inherent in LLM inference, characterized by a power-law
  distribution in neuron activation. This distribution indicates that a small
  subset of neurons, termed hot neurons, are consistently activated across
  inputs, while the majority, cold neurons, vary based on specific inputs.
  PowerInfer exploits such an insight to design a GPU-CPU hybrid inference
  engine: hot-activated neurons are preloaded onto the GPU for fast access, while
  cold-activated neurons are computed on the CPU, thus significantly reducing GPU
  memory demands and CPU-GPU data transfers. PowerInfer further integrates
  adaptive predictors and neuron-aware sparse operators, optimizing the
  efficiency of neuron activation and computational sparsity. Evaluation shows
  that PowerInfer attains an average token generation rate of 13.20 tokens/s,
  with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a
  single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier
  server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x
  while retaining model accuracy.
paper_authors._id: 6583b1581804c2d060ac9e49
paper_authors.name: Yixin Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:40:50.057Z
paper_authors._id: 642924f916d4d8293c93af08
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642924f916d4d8293c93af08/Olp4zFPgGu2E6mFFPXfQO.jpeg
paper_authors.fullname: Yixin Song
paper_authors.type: user
paper_authors.user: yixinsong
paper_authors._id: 6583b1581804c2d060ac9e4a
paper_authors.name: Zeyu Mi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:41:10.655Z
paper_authors._id: 6565a74575968d50cc1e51d3
paper_authors.avatarUrl: /avatars/cc50e33b4e275aff2c017b7a2d7830e4.svg
paper_authors.fullname: Zeyu Mi
paper_authors.type: user
paper_authors.user: yzmizeyu
paper_authors._id: 6583b1581804c2d060ac9e4b
paper_authors.name: Haotong Xie
paper_authors._id: 6583b1581804c2d060ac9e4c
paper_authors.name: Haibo Chen
paper_title: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
paper_summary: This paper introduces PowerInfer, a high-speed Large Language Model (LLM)
  inference engine on a personal computer (PC) equipped with a single
  consumer-grade GPU. The key underlying the design of PowerInfer is exploiting
  the high locality inherent in LLM inference, characterized by a power-law
  distribution in neuron activation. This distribution indicates that a small
  subset of neurons, termed hot neurons, are consistently activated across
  inputs, while the majority, cold neurons, vary based on specific inputs.
  PowerInfer exploits such an insight to design a GPU-CPU hybrid inference
  engine: hot-activated neurons are preloaded onto the GPU for fast access, while
  cold-activated neurons are computed on the CPU, thus significantly reducing GPU
  memory demands and CPU-GPU data transfers. PowerInfer further integrates
  adaptive predictors and neuron-aware sparse operators, optimizing the
  efficiency of neuron activation and computational sparsity. Evaluation shows
  that PowerInfer attains an average token generation rate of 13.20 tokens/s,
  with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a
  single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier
  server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x
  while retaining model accuracy.
paper_ai_keywords: Large Language Model (LLM) GPU-CPU hybrid inference hot neurons cold neurons adaptive predictors neuron-aware sparse operators token generation rate OPT-175B NVIDIA RTX 4090 GPU A100 GPU llama.cpp model accuracy