title: WINA: Weight Informed Neuron Activation for Accelerating Large Language
    Model Inference
summary: The growing computational demands of large language models (LLMs) make
  efficient inference and activation strategies increasingly critical. While
  recent approaches, such as Mixture-of-Experts (MoE), leverage selective
  activation but require specialized training, training-free sparse activation
  methods offer broader applicability and superior resource efficiency through
  their plug-and-play design. However, many existing methods rely solely on
  hidden state magnitudes to determine activation, resulting in high
  approximation errors and suboptimal inference accuracy. To address these
  limitations, we propose WINA (Weight Informed Neuron Activation), a novel,
  simple, and training-free sparse activation framework that jointly considers
  hidden state magnitudes and the column-wise ell_2-norms of weight matrices.
  We show that this leads to a sparsification strategy that obtains optimal
  approximation error bounds with theoretical guarantees tighter than existing
  techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g.,
  TEAL) by up to 2.94% in average performance at the same sparsity levels,
  across a diverse set of LLM architectures and datasets. These results position
  WINA as a new performance frontier for training-free sparse activation in LLM
  inference, advancing training-free sparse activation methods and setting a
  robust baseline for efficient inference. The source code is available at
  https://github.com/microsoft/wina.
paper_authors._id: 683525ac1c31d709ba52273e
paper_authors.name: Sihan Chen
paper_authors._id: 683525ac1c31d709ba52273f
paper_authors.name: Dan Zhao
paper_authors._id: 683525ac1c31d709ba522740
paper_authors.name: Jongwoo Ko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:00:10.956Z
paper_authors._id: 64b7628af902508f0d7ae112
paper_authors.avatarUrl: /avatars/83c155254486e80c1dfd14676fdf9215.svg
paper_authors.fullname: Jongwoo Ko
paper_authors.type: user
paper_authors.user: jongwooko
paper_authors._id: 683525ac1c31d709ba522741
paper_authors.name: Colby Banbury
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T13:00:04.107Z
paper_authors._id: 6256fd8c8c6638f8a93baf05
paper_authors.avatarUrl: /avatars/2cb5b84c88aa07418beaeae5526c86a8.svg
paper_authors.fullname: Colby Banbury
paper_authors.type: user
paper_authors.user: colbybanbury
paper_authors._id: 683525ac1c31d709ba522742
paper_authors.name: Huiping Zhuang
paper_authors._id: 683525ac1c31d709ba522743
paper_authors.name: Luming Liang
paper_authors._id: 683525ac1c31d709ba522744
paper_authors.name: Tianyi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:51:46.088Z
paper_authors._id: 64ad94f05a4a60156925ec96
paper_authors.avatarUrl: /avatars/643bdb076e703bfcc89cec6fccb756c6.svg
paper_authors.fullname: Tianyi Chen
paper_authors.type: user
paper_authors.user: tianyic
paper_title: WINA: Weight Informed Neuron Activation for Accelerating Large Language
    Model Inference
paper_summary: The growing computational demands of large language models (LLMs) make
  efficient inference and activation strategies increasingly critical. While
  recent approaches, such as Mixture-of-Experts (MoE), leverage selective
  activation but require specialized training, training-free sparse activation
  methods offer broader applicability and superior resource efficiency through
  their plug-and-play design. However, many existing methods rely solely on
  hidden state magnitudes to determine activation, resulting in high
  approximation errors and suboptimal inference accuracy. To address these
  limitations, we propose WINA (Weight Informed Neuron Activation), a novel,
  simple, and training-free sparse activation framework that jointly considers
  hidden state magnitudes and the column-wise ell_2-norms of weight matrices.
  We show that this leads to a sparsification strategy that obtains optimal
  approximation error bounds with theoretical guarantees tighter than existing
  techniques. Empirically, WINA also outperforms state-of-the-art methods (e.g.,
  TEAL) by up to 2.94% in average performance at the same sparsity levels,
  across a diverse set of LLM architectures and datasets. These results position
  WINA as a new performance frontier for training-free sparse activation in LLM
  inference, advancing training-free sparse activation methods and setting a
  robust baseline for efficient inference. The source code is available at
  https://github.com/microsoft/wina.
paper_ai_summary: WINA, a training-free sparse activation framework for large language models, improves inference accuracy by considering hidden state magnitudes and weight matrix norms, outperforming existing methods.
paper_ai_keywords: Mixture-of-Experts (MoE) sparse activation hidden state magnitudes column-wise $\ell_2$-norms weight matrices sparsification strategy approximation error bounds training-free sparse activation large language models