title: A Refined Analysis of Massive Activations in LLMs
summary: Motivated in part by their relevance for low-precision training and
  quantization, massive activations in large language models (LLMs) have recently
  emerged as a topic of interest. However, existing analyses are limited in
  scope, and generalizability across architectures is unclear. This paper helps
  address some of these gaps by conducting an analysis of massive activations
  across a broad range of LLMs, including both GLU-based and non-GLU-based
  architectures. Our findings challenge several prior assumptions, most
  importantly: (1) not all massive activations are detrimental, i.e. suppressing
  them does not lead to an explosion of perplexity or a collapse in downstream
  task performance; (2) proposed mitigation strategies such as Attention KV bias
  are model-specific and ineffective in certain cases. We consequently
  investigate novel hybrid mitigation strategies; in particular pairing Target
  Variance Rescaling (TVR) with Attention KV bias or Dynamic Tanh (DyT)
  successfully balances the mitigation of massive activations with preserved
  downstream model performance in the scenarios we investigated. Our code is
  available at: https://github.com/bluorion-com/refine_massive_activations.
paper_authors._id: 67ea01e3d13d75fc155fa69d
paper_authors.name: Louis Owen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:11:39.976Z
paper_authors._id: 6071c4b270e11b30cfcfd7a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6071c4b270e11b30cfcfd7a3/-1ekCBzSTpqxkkul0bgmI.jpeg
paper_authors.fullname: Louis Owen
paper_authors.type: user
paper_authors.user: louisowen6
paper_authors._id: 67ea01e3d13d75fc155fa69e
paper_authors.name: Nilabhra Roy Chowdhury
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:11:37.938Z
paper_authors._id: 645a0d3dd6648853107c5fdc
paper_authors.avatarUrl: /avatars/1e3b6a4f5ce81a707ba7cbdf81631091.svg
paper_authors.fullname: Nilabhra Roy Chowdhury
paper_authors.type: user
paper_authors.user: nilabhra
paper_authors._id: 67ea01e3d13d75fc155fa69f
paper_authors.name: Abhay Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:11:35.757Z
paper_authors._id: 62cd4b03c5cc157be82f0b56
paper_authors.avatarUrl: /avatars/351e963c1c763d507ae78cbcd62966a3.svg
paper_authors.fullname: Abhay kumar
paper_authors.type: user
paper_authors.user: akanyaani
paper_authors._id: 67ea01e3d13d75fc155fa6a0
paper_authors.name: Fabian Güra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T13:33:11.338Z
paper_authors._id: 65e4be59e8b017ee1310a1b6
paper_authors.avatarUrl: /avatars/c3f7cdf5d0859cb80bfb2b970a675dfa.svg
paper_authors.fullname: Fabian
paper_authors.type: user
paper_authors.user: gueraf
paper_title: A Refined Analysis of Massive Activations in LLMs
paper_summary: Motivated in part by their relevance for low-precision training and
  quantization, massive activations in large language models (LLMs) have recently
  emerged as a topic of interest. However, existing analyses are limited in
  scope, and generalizability across architectures is unclear. This paper helps
  address some of these gaps by conducting an analysis of massive activations
  across a broad range of LLMs, including both GLU-based and non-GLU-based
  architectures. Our findings challenge several prior assumptions, most
  importantly: (1) not all massive activations are detrimental, i.e. suppressing
  them does not lead to an explosion of perplexity or a collapse in downstream
  task performance; (2) proposed mitigation strategies such as Attention KV bias
  are model-specific and ineffective in certain cases. We consequently
  investigate novel hybrid mitigation strategies; in particular pairing Target
  Variance Rescaling (TVR) with Attention KV bias or Dynamic Tanh (DyT)
  successfully balances the mitigation of massive activations with preserved
  downstream model performance in the scenarios we investigated. Our code is
  available at: https://github.com/bluorion-com/refine_massive_activations.
paper_ai_summary: An analysis of massive activations in large language models shows that suppression is not always detrimental and proposes hybrid mitigation strategies that balance activation management with model performance.
paper_ai_keywords: low-precision training quantization massive activations large language models LLMs GLU-based non-GLU-based perplexity downstream task performance Attention KV bias Target Variance Rescaling TVR Dynamic Tanh DyT