title: Sparsing Law: Towards Large Language Models with Greater Activation
    Sparsity
summary: Activation sparsity denotes the existence of substantial weakly-contributed
  elements within activation outputs that can be eliminated, benefiting many
  important applications concerned with large language models (LLMs). Although
  promoting greater activation sparsity within LLMs deserves deep studies,
  existing works lack comprehensive and quantitative research on the correlation
  between activation sparsity and potentially influential factors. In this paper,
  we present a comprehensive study on the quantitative scaling properties and
  influential factors of the activation sparsity within decoder-only
  Transformer-based LLMs. Specifically, we propose PPL-p% sparsity, a precise
  and performance-aware activation sparsity metric that is applicable to any
  activation function. Through extensive experiments, we find several important
  phenomena. Firstly, different activation functions exhibit comparable
  performance but opposite training-time sparsity trends. The activation ratio
  (i.e., 1-sparsity ratio) evolves as a convergent increasing
  power-law and decreasing logspace power-law with the amount of training data
  for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate
  that ReLU is more efficient as the activation function than SiLU and can
  leverage more training data to improve activation sparsity. Secondly, the
  activation ratio linearly increases with the width-depth ratio below a certain
  bottleneck point, indicating the potential advantage of a deeper architecture
  at a fixed parameter scale. Finally, at similar width-depth ratios, we
  surprisingly find that the limit value of activation sparsity varies weakly
  with the parameter scale, i.e., the activation patterns within LLMs are
  insensitive to the parameter scale. These empirical laws towards LLMs with
  greater activation sparsity have important implications for making LLMs more
  efficient and interpretable.
paper_authors._id: 6729a58ae53061b3dc4bb379
paper_authors.name: Yuqi Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:38:17.245Z
paper_authors._id: 65f2f8bfcec22d29ce4bd23f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f2f8bfcec22d29ce4bd23f/WGM__qyGv2PfbPg-U7T-P.jpeg
paper_authors.fullname: luoyuqi
paper_authors.type: user
paper_authors.user: demerzel-iv
paper_authors._id: 6729a58ae53061b3dc4bb37a
paper_authors.name: Chenyang Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:43:14.625Z
paper_authors._id: 64c09684e56520a63d35ec87
paper_authors.avatarUrl: /avatars/86a338e8d122a66a94143bbb9bf3ebf8.svg
paper_authors.fullname: Chenyang Song
paper_authors.type: user
paper_authors.user: Raincleared
paper_authors._id: 6729a58ae53061b3dc4bb37b
paper_authors.name: Xu Han
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-05T04:57:20.885Z
paper_authors._id: 65b8bd1dd4f282b64711d049
paper_authors.avatarUrl: /avatars/4e507f0ef4f3420bcff2677fd8ee14e5.svg
paper_authors.fullname: Xu Han
paper_authors.type: user
paper_authors.user: SillyXu
paper_authors._id: 6729a58ae53061b3dc4bb37c
paper_authors.name: Yingfa Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:43:08.749Z
paper_authors._id: 6144e4667f2544bb450787b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
paper_authors.fullname: Yingfa Chen
paper_authors.type: user
paper_authors.user: chen-yingfa
paper_authors._id: 6729a58ae53061b3dc4bb37d
paper_authors.name: Chaojun Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:43:01.749Z
paper_authors._id: 608f6d72283d0a8d7be9d1f9
paper_authors.avatarUrl: /avatars/7f499a37019359a3c488ba6cc11751fc.svg
paper_authors.fullname: Chaojun XIAO
paper_authors.type: user
paper_authors.user: xcjthu
paper_authors._id: 6729a58ae53061b3dc4bb37e
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:56:15.167Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 6729a58ae53061b3dc4bb37f
paper_authors.name: Maosong Sun
paper_title: Sparsing Law: Towards Large Language Models with Greater Activation
    Sparsity
paper_summary: Activation sparsity denotes the existence of substantial weakly-contributed
  elements within activation outputs that can be eliminated, benefiting many
  important applications concerned with large language models (LLMs). Although
  promoting greater activation sparsity within LLMs deserves deep studies,
  existing works lack comprehensive and quantitative research on the correlation
  between activation sparsity and potentially influential factors. In this paper,
  we present a comprehensive study on the quantitative scaling properties and
  influential factors of the activation sparsity within decoder-only
  Transformer-based LLMs. Specifically, we propose PPL-p% sparsity, a precise
  and performance-aware activation sparsity metric that is applicable to any
  activation function. Through extensive experiments, we find several important
  phenomena. Firstly, different activation functions exhibit comparable
  performance but opposite training-time sparsity trends. The activation ratio
  (i.e., 1-sparsity ratio) evolves as a convergent increasing
  power-law and decreasing logspace power-law with the amount of training data
  for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate
  that ReLU is more efficient as the activation function than SiLU and can
  leverage more training data to improve activation sparsity. Secondly, the
  activation ratio linearly increases with the width-depth ratio below a certain
  bottleneck point, indicating the potential advantage of a deeper architecture
  at a fixed parameter scale. Finally, at similar width-depth ratios, we
  surprisingly find that the limit value of activation sparsity varies weakly
  with the parameter scale, i.e., the activation patterns within LLMs are
  insensitive to the parameter scale. These empirical laws towards LLMs with
  greater activation sparsity have important implications for making LLMs more
  efficient and interpretable.
paper_ai_summary: A comprehensive study of activation sparsity in decoder-only Transformer-based LLMs reveals relationships between sparsity, activation functions, training data, architecture, and parameter scales, offering insights to enhance model efficiency and interpretability.
paper_ai_keywords: activation sparsity PPL-$p\%$ sparsity activation functions SiLU ReLU activation ratio power-law convergent increasing power-law decreasing logspace power-law width-depth ratio parameter scale