Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
We introduce, Q-Sparse, a simple yet effective approach to training
sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity
of activations in LLMs which can bring significant efficiency gains in
inference. This is achieved by applying top-K sparsification to the activations
and the straight-through-estimator to the training. The key results from this
work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs
while being much more efficient at inference time; (2) We present an
inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is
effective in different settings, including training-from-scratch,
continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for
both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the
synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the
cornerstone and a clear path to revolutionize the efficiency, including cost
and energy consumption, of future LLMs.
669615f637b573b6007c5224
Hongyu Wang
claimed_verified
2024-07-16T20:15:17.326Z
63f71771d36951307fcb4dcd
https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
Hongyu Wang
user
hongyuw
669615f637b573b6007c5225
Shuming Ma
669615f637b573b6007c5226
Ruiping Wang
669615f637b573b6007c5227
Furu Wei
Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
We introduce, Q-Sparse, a simple yet effective approach to training
sparsely-activated large language models (LLMs). Q-Sparse enables full sparsity
of activations in LLMs which can bring significant efficiency gains in
inference. This is achieved by applying top-K sparsification to the activations
and the straight-through-estimator to the training. The key results from this
work are, (1) Q-Sparse can achieve results comparable to those of baseline LLMs
while being much more efficient at inference time; (2) We present an
inference-optimal scaling law for sparsely-activated LLMs; (3) Q-Sparse is
effective in different settings, including training-from-scratch,
continue-training of off-the-shelf LLMs, and finetuning; (4) Q-Sparse works for
both full-precision and 1-bit LLMs (e.g., BitNet b1.58). Particularly, the
synergy of BitNet b1.58 and Q-Sparse (can be equipped with MoE) provides the
cornerstone and a clear path to revolutionize the efficiency, including cost
and energy consumption, of future LLMs.
Q-Sparse enhances sparsely-activated large language models for efficient inference across various training and deployment scenarios, using top-K sparsification and straight-through-estimators.
sparsely-activated large language models Q-Sparse top-K sparsification straight-through-estimator inference-optimal scaling law training-from-scratch continue-training finetuning BitNet b1.58 MoE