Sparsing Law: Towards Large Language Models with Greater Activation
  Sparsity
Activation sparsity denotes the existence of substantial weakly-contributed
elements within activation outputs that can be eliminated, benefiting many
important applications concerned with large language models (LLMs). Although
promoting greater activation sparsity within LLMs deserves deep studies,
existing works lack comprehensive and quantitative research on the correlation
between activation sparsity and potentially influential factors. In this paper,
we present a comprehensive study on the quantitative scaling properties and
influential factors of the activation sparsity within decoder-only
Transformer-based LLMs. Specifically, we propose PPL-p% sparsity, a precise
and performance-aware activation sparsity metric that is applicable to any
activation function. Through extensive experiments, we find several important
phenomena. Firstly, different activation functions exhibit comparable
performance but opposite training-time sparsity trends. The activation ratio
(i.e., 1-sparsity ratio) evolves as a convergent increasing
power-law and decreasing logspace power-law with the amount of training data
for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate
that ReLU is more efficient as the activation function than SiLU and can
leverage more training data to improve activation sparsity. Secondly, the
activation ratio linearly increases with the width-depth ratio below a certain
bottleneck point, indicating the potential advantage of a deeper architecture
at a fixed parameter scale. Finally, at similar width-depth ratios, we
surprisingly find that the limit value of activation sparsity varies weakly
with the parameter scale, i.e., the activation patterns within LLMs are
insensitive to the parameter scale. These empirical laws towards LLMs with
greater activation sparsity have important implications for making LLMs more
efficient and interpretable.
6729a58ae53061b3dc4bb379
Yuqi Luo
claimed_verified
2024-11-15T09:38:17.245Z
65f2f8bfcec22d29ce4bd23f
https://cdn-avatars.huggingface.co/v1/production/uploads/65f2f8bfcec22d29ce4bd23f/WGM__qyGv2PfbPg-U7T-P.jpeg
luoyuqi
user
demerzel-iv
6729a58ae53061b3dc4bb37a
Chenyang Song
admin_assigned
2024-11-05T14:43:14.625Z
64c09684e56520a63d35ec87
/avatars/86a338e8d122a66a94143bbb9bf3ebf8.svg
Chenyang Song
user
Raincleared
6729a58ae53061b3dc4bb37b
Xu Han
extracted_pending
2024-11-05T04:57:20.885Z
65b8bd1dd4f282b64711d049
/avatars/4e507f0ef4f3420bcff2677fd8ee14e5.svg
Xu Han
user
SillyXu
6729a58ae53061b3dc4bb37c
Yingfa Chen
admin_assigned
2024-11-05T14:43:08.749Z
6144e4667f2544bb450787b2
https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
Yingfa Chen
user
chen-yingfa
6729a58ae53061b3dc4bb37d
Chaojun Xiao
admin_assigned
2024-11-05T14:43:01.749Z
608f6d72283d0a8d7be9d1f9
/avatars/7f499a37019359a3c488ba6cc11751fc.svg
Chaojun XIAO
user
xcjthu
6729a58ae53061b3dc4bb37e
Zhiyuan Liu
claimed_verified
2025-06-02T07:56:15.167Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
6729a58ae53061b3dc4bb37f
Maosong Sun
Sparsing Law: Towards Large Language Models with Greater Activation
  Sparsity
Activation sparsity denotes the existence of substantial weakly-contributed
elements within activation outputs that can be eliminated, benefiting many
important applications concerned with large language models (LLMs). Although
promoting greater activation sparsity within LLMs deserves deep studies,
existing works lack comprehensive and quantitative research on the correlation
between activation sparsity and potentially influential factors. In this paper,
we present a comprehensive study on the quantitative scaling properties and
influential factors of the activation sparsity within decoder-only
Transformer-based LLMs. Specifically, we propose PPL-p% sparsity, a precise
and performance-aware activation sparsity metric that is applicable to any
activation function. Through extensive experiments, we find several important
phenomena. Firstly, different activation functions exhibit comparable
performance but opposite training-time sparsity trends. The activation ratio
(i.e., 1-sparsity ratio) evolves as a convergent increasing
power-law and decreasing logspace power-law with the amount of training data
for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate
that ReLU is more efficient as the activation function than SiLU and can
leverage more training data to improve activation sparsity. Secondly, the
activation ratio linearly increases with the width-depth ratio below a certain
bottleneck point, indicating the potential advantage of a deeper architecture
at a fixed parameter scale. Finally, at similar width-depth ratios, we
surprisingly find that the limit value of activation sparsity varies weakly
with the parameter scale, i.e., the activation patterns within LLMs are
insensitive to the parameter scale. These empirical laws towards LLMs with
greater activation sparsity have important implications for making LLMs more
efficient and interpretable.
A comprehensive study of activation sparsity in decoder-only Transformer-based LLMs reveals relationships between sparsity, activation functions, training data, architecture, and parameter scales, offering insights to enhance model efficiency and interpretability.
activation sparsity PPL-$p\%$ sparsity activation functions SiLU ReLU activation ratio power-law convergent increasing power-law decreasing logspace power-law width-depth ratio parameter scale