title: ReLU's Revival: On the Entropic Overload in Normalization-Free Large
    Language Models
summary: LayerNorm is a critical component in modern large language models (LLMs) for
  stabilizing training and ensuring smooth optimization. However, it introduces
  significant challenges in mechanistic interpretability, outlier feature
  suppression, faithful signal propagation, and computational and communication
  complexity of private inference. This work explores desirable activation
  functions in normalization-free decoder-only LLMs. Contrary to the conventional
  preference for the GELU in transformer-based models, our empirical findings
  demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in
  LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We
  discover a key issue with GELU, where early layers experience entropic
  overload, leading to the under-utilization of the representational capacity of
  attention heads. This highlights that smoother activations like GELU are {\em
  ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical
  properties -- specialization in input space and intra-class selectivity -- lead
  to improved learning dynamics and better information retention in the absence
  of LayerNorm. This study offers key insights for optimizing transformer
  architectures where LayerNorm introduces significant challenges.
paper_authors._id: 670ec609aeee51ca904b26a0
paper_authors.name: Nandan Kumar Jha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T21:17:50.453Z
paper_authors._id: 670ec3f6db1a6bcfe832e0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-mZNLeLJoXzkwPgYO38lF.png
paper_authors.fullname: Nandan Kumar Jha
paper_authors.type: user
paper_authors.user: nandan523
paper_authors._id: 670ec609aeee51ca904b26a1
paper_authors.name: Brandon Reagen
paper_title: ReLU's Revival: On the Entropic Overload in Normalization-Free Large
    Language Models
paper_summary: LayerNorm is a critical component in modern large language models (LLMs) for
  stabilizing training and ensuring smooth optimization. However, it introduces
  significant challenges in mechanistic interpretability, outlier feature
  suppression, faithful signal propagation, and computational and communication
  complexity of private inference. This work explores desirable activation
  functions in normalization-free decoder-only LLMs. Contrary to the conventional
  preference for the GELU in transformer-based models, our empirical findings
  demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in
  LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We
  discover a key issue with GELU, where early layers experience entropic
  overload, leading to the under-utilization of the representational capacity of
  attention heads. This highlights that smoother activations like GELU are {\em
  ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical
  properties -- specialization in input space and intra-class selectivity -- lead
  to improved learning dynamics and better information retention in the absence
  of LayerNorm. This study offers key insights for optimizing transformer
  architectures where LayerNorm introduces significant challenges.
paper_ai_summary: ReLU outperforms GELU in LayerNorm-free transformer models, improving perplexity and learning dynamics.
paper_ai_keywords: LayerNorm large language models normalization-free decoder-only transformer-based models GELU ReLU entropic overload representational capacity attention heads learning dynamics information retention