WINA: Weight Informed Neuron Activation for Accelerating Large Language
  Model Inference
The growing computational demands of large language models (LLMs) make
efficient inference and activation strategies increasingly critical. While
recent approaches, such as Mixture-of-Experts (MoE), leverage selective
activation but require specialized training, training-free sparse activation
methods offer broader applicability and superior resource efficiency through
their plug-and-play design. However, many existing methods rely solely on
hidden state magnitudes to determine activation, resulting in high
approximation errors and suboptimal inference accuracy. To address these
limitations, we propose WINA (Weight Informed Neuron Activation), a novel,
simple, and training-free sparse activation framework that jointly considers
hidden state magnitudes and the column-wise ell_2-norms of weight matrices.
We show that this leads to a sparsification strategy that obtains optimal
approximation error bounds with theoretical guarantees tighter than existing
techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g.,
TEAL) by up to 2.94% in average performance at the same sparsity levels,
across a diverse set of LLM architectures and datasets. These results position
WINA as a new performance frontier for training-free sparse activation in LLM
inference, advancing training-free sparse activation methods and setting a
robust baseline for efficient inference. The source code is available at
https://github.com/microsoft/wina.
683525ac1c31d709ba52273e
Sihan Chen
683525ac1c31d709ba52273f
Dan Zhao
683525ac1c31d709ba522740
Jongwoo Ko
admin_assigned
2025-05-27T13:00:10.956Z
64b7628af902508f0d7ae112
/avatars/83c155254486e80c1dfd14676fdf9215.svg
Jongwoo Ko
user
jongwooko
683525ac1c31d709ba522741
Colby Banbury
admin_assigned
2025-05-27T13:00:04.107Z
6256fd8c8c6638f8a93baf05
/avatars/2cb5b84c88aa07418beaeae5526c86a8.svg
Colby Banbury
user
colbybanbury
683525ac1c31d709ba522742
Huiping Zhuang
claimed_verified
2025-06-24T08:11:27.452Z
64840f6c4a01519bcbceaf7c
/avatars/9f3d08a2672fcaaf139d2e4b419eedbb.svg
ZHUANGHP
user
ZHUANGHP
683525ac1c31d709ba522743
Luming Liang
683525ac1c31d709ba522744
Tianyi Chen
claimed_verified
2025-05-27T07:51:46.088Z
64ad94f05a4a60156925ec96
/avatars/643bdb076e703bfcc89cec6fccb756c6.svg
Tianyi Chen
user
tianyic
WINA: Weight Informed Neuron Activation for Accelerating Large Language
  Model Inference
The growing computational demands of large language models (LLMs) make
efficient inference and activation strategies increasingly critical. While
recent approaches, such as Mixture-of-Experts (MoE), leverage selective
activation but require specialized training, training-free sparse activation
methods offer broader applicability and superior resource efficiency through
their plug-and-play design. However, many existing methods rely solely on
hidden state magnitudes to determine activation, resulting in high
approximation errors and suboptimal inference accuracy. To address these
limitations, we propose WINA (Weight Informed Neuron Activation), a novel,
simple, and training-free sparse activation framework that jointly considers
hidden state magnitudes and the column-wise ell_2-norms of weight matrices.
We show that this leads to a sparsification strategy that obtains optimal
approximation error bounds with theoretical guarantees tighter than existing
techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g.,
TEAL) by up to 2.94% in average performance at the same sparsity levels,
across a diverse set of LLM architectures and datasets. These results position
WINA as a new performance frontier for training-free sparse activation in LLM
inference, advancing training-free sparse activation methods and setting a
robust baseline for efficient inference. The source code is available at
https://github.com/microsoft/wina.
WINA, a training-free sparse activation framework for large language models, improves inference accuracy by considering hidden state magnitudes and weight matrix norms, outperforming existing methods.
Mixture-of-Experts (MoE) sparse activation hidden state magnitudes column-wise $\ell_2$-norms weight matrices sparsification strategy approximation error bounds training-free sparse activation large language models