title: Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated
    Parameters
summary: Exploiting activation sparsity is a promising approach to significantly
  accelerating the inference process of large language models (LLMs) without
  compromising performance. However, activation sparsity is determined by
  activation functions, and commonly used ones like SwiGLU and GeGLU exhibit
  limited sparsity. Simply replacing these functions with ReLU fails to achieve
  sufficient sparsity. Moreover, inadequate training data can further increase
  the risk of performance degradation. To address these challenges, we propose a
  novel dReLU function, which is designed to improve LLM activation sparsity,
  along with a high-quality training data mixture ratio to facilitate effective
  sparsification. Additionally, we leverage sparse activation patterns within the
  Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to
  further boost efficiency. By applying our neuron sparsification method to the
  Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are
  activated per inference iteration, respectively, while achieving even more
  powerful model performance. Evaluation results demonstrate that this sparsity
  achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our
  TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second.
  Our models are available at https://huggingface.co/PowerInfer
paper_authors._id: 666a75d93536e9a956c5960c
paper_authors.name: Yixin Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T13:46:44.174Z
paper_authors._id: 642924f916d4d8293c93af08
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642924f916d4d8293c93af08/Olp4zFPgGu2E6mFFPXfQO.jpeg
paper_authors.fullname: Yixin Song
paper_authors.type: user
paper_authors.user: yixinsong
paper_authors._id: 666a75d93536e9a956c5960d
paper_authors.name: Haotong Xie
paper_authors._id: 666a75d93536e9a956c5960e
paper_authors.name: Zhengyan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:03:31.823Z
paper_authors._id: 65654ed2219af7f841640f27
paper_authors.avatarUrl: /avatars/e6904b3479fc5e65ea1f752919ca8290.svg
paper_authors.fullname: Zhengyan Zhang
paper_authors.type: user
paper_authors.user: ZhengyanZhang
paper_authors._id: 666a75d93536e9a956c5960f
paper_authors.name: Bo Wen
paper_authors._id: 666a75d93536e9a956c59610
paper_authors.name: Li Ma
paper_authors._id: 666a75d93536e9a956c59611
paper_authors.name: Zeyu Mi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:30:32.510Z
paper_authors._id: 6565a74575968d50cc1e51d3
paper_authors.avatarUrl: /avatars/cc50e33b4e275aff2c017b7a2d7830e4.svg
paper_authors.fullname: Zeyu Mi
paper_authors.type: user
paper_authors.user: yzmizeyu
paper_authors._id: 666a75d93536e9a956c59612
paper_authors.name: Haibo Chen
paper_title: Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated
    Parameters
paper_summary: Exploiting activation sparsity is a promising approach to significantly
  accelerating the inference process of large language models (LLMs) without
  compromising performance. However, activation sparsity is determined by
  activation functions, and commonly used ones like SwiGLU and GeGLU exhibit
  limited sparsity. Simply replacing these functions with ReLU fails to achieve
  sufficient sparsity. Moreover, inadequate training data can further increase
  the risk of performance degradation. To address these challenges, we propose a
  novel dReLU function, which is designed to improve LLM activation sparsity,
  along with a high-quality training data mixture ratio to facilitate effective
  sparsification. Additionally, we leverage sparse activation patterns within the
  Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to
  further boost efficiency. By applying our neuron sparsification method to the
  Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are
  activated per inference iteration, respectively, while achieving even more
  powerful model performance. Evaluation results demonstrate that this sparsity
  achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our
  TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second.
  Our models are available at https://huggingface.co/PowerInfer
paper_ai_keywords: activation sparsity SwiGLU GeGLU ReLU dReLU Feed-Forward Network (FFN) Mixture-of-Experts (MoE) neuron sparsification Mistral Mixtral decoding speedup inference speed tokens per second