title: Transformers without Normalization
summary: Normalization layers are ubiquitous in modern neural networks and have long
  been considered essential. This work demonstrates that Transformers without
  normalization can achieve the same or better performance using a remarkably
  simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation
  DyT(x) = tanh(alpha x), as a drop-in replacement for normalization
  layers in Transformers. DyT is inspired by the observation that layer
  normalization in Transformers often produces tanh-like, S-shaped input-output
  mappings. By incorporating DyT, Transformers without normalization can match or
  exceed the performance of their normalized counterparts, mostly without
  hyperparameter tuning. We validate the effectiveness of Transformers with DyT
  across diverse settings, ranging from recognition to generation, supervised to
  self-supervised learning, and computer vision to language models. These
  findings challenge the conventional understanding that normalization layers are
  indispensable in modern neural networks, and offer new insights into their role
  in deep networks.
paper_authors._id: 67d3b0a87443e648e8aa1ea6
paper_authors.name: Jiachen Zhu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-14T05:13:31.648Z
paper_authors._id: 6552126dd8a8835b66653767
paper_authors.avatarUrl: /avatars/0b1dad9ebaeada8f5e7ebe453123960b.svg
paper_authors.fullname: Jiachen Zhu
paper_authors.type: user
paper_authors.user: JiachenZhu
paper_authors._id: 67d3b0a87443e648e8aa1ea7
paper_authors.name: Xinlei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:45:29.927Z
paper_authors._id: 63e58e3a006a775275e59e41
paper_authors.avatarUrl: /avatars/75262a35b27a2ae1939df9118120d99e.svg
paper_authors.fullname: Xinlei Chen
paper_authors.type: user
paper_authors.user: endernewton
paper_authors._id: 67d3b0a87443e648e8aa1ea8
paper_authors.name: Kaiming He
paper_authors._id: 67d3b0a87443e648e8aa1ea9
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:45:43.974Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 67d3b0a87443e648e8aa1eaa
paper_authors.name: Zhuang Liu
paper_title: Transformers without Normalization
paper_summary: Normalization layers are ubiquitous in modern neural networks and have long
  been considered essential. This work demonstrates that Transformers without
  normalization can achieve the same or better performance using a remarkably
  simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation
  DyT(x) = tanh(alpha x), as a drop-in replacement for normalization
  layers in Transformers. DyT is inspired by the observation that layer
  normalization in Transformers often produces tanh-like, S-shaped input-output
  mappings. By incorporating DyT, Transformers without normalization can match or
  exceed the performance of their normalized counterparts, mostly without
  hyperparameter tuning. We validate the effectiveness of Transformers with DyT
  across diverse settings, ranging from recognition to generation, supervised to
  self-supervised learning, and computer vision to language models. These
  findings challenge the conventional understanding that normalization layers are
  indispensable in modern neural networks, and offer new insights into their role
  in deep networks.
paper_ai_summary: Dynamic Tanh (DyT) replaces normalization layers in Transformers, achieving equivalent or superior performance without hyperparameter tuning across various tasks.
paper_ai_keywords: Normalization layers Dynamic Tanh DyT layer normalization Transformers input-output mappings hyperparameter tuning recognition generation supervised learning self-supervised learning computer vision language models