title: Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
summary: We introduce, Q-Sparse, a simple yet effective approach to training
  sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity
  of activations in LLMs which can bring significant efficiency gains in
  inference. This is achieved by applying top-K sparsification to the activations
  and the straight-through-estimator to the training. The key results from this
  work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs
  while being much more efficient at inference time; (2) We present an
  inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is
  effective in different settings, including training-from-scratch,
  continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for
  both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the
  synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the
  cornerstone and a clear path to revolutionize the efficiency, including cost
  and energy consumption, of future LLMs.
paper_authors._id: 669615f637b573b6007c5224
paper_authors.name: Hongyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-16T20:15:17.326Z
paper_authors._id: 63f71771d36951307fcb4dcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
paper_authors.fullname: Hongyu Wang
paper_authors.type: user
paper_authors.user: hongyuw
paper_authors._id: 669615f637b573b6007c5225
paper_authors.name: Shuming Ma
paper_authors._id: 669615f637b573b6007c5226
paper_authors.name: Ruiping Wang
paper_authors._id: 669615f637b573b6007c5227
paper_authors.name: Furu Wei
paper_title: Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
paper_summary: We introduce, Q-Sparse, a simple yet effective approach to training
  sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity
  of activations in LLMs which can bring significant efficiency gains in
  inference. This is achieved by applying top-K sparsification to the activations
  and the straight-through-estimator to the training. The key results from this
  work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs
  while being much more efficient at inference time; (2) We present an
  inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is
  effective in different settings, including training-from-scratch,
  continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for
  both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the
  synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the
  cornerstone and a clear path to revolutionize the efficiency, including cost
  and energy consumption, of future LLMs.
paper_ai_summary: Q-Sparse enhances sparsely-activated large language models for efficient inference across various training and deployment scenarios, using top-K sparsification and straight-through-estimators.
paper_ai_keywords: sparsely-activated large language models Q-Sparse top-K sparsification straight-through-estimator inference-optimal scaling law training-from-scratch continue-training finetuning BitNet b1.58 MoE