title: The Curse of Depth in Large Language Models
summary: In this paper, we introduce the Curse of Depth, a concept that highlights,
  explains, and addresses the recent observation in modern Large Language
  Models(LLMs) where nearly half of the layers are less effective than expected.
  We first confirm the wide existence of this phenomenon across the most popular
  families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis,
  theoretically and empirically, identifies that the underlying reason for the
  ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer
  Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer
  LLMs, its output variance exponentially grows with the model depth, which
  undesirably causes the derivative of the deep Transformer blocks to be an
  identity matrix, and therefore barely contributes to the training. To resolve
  this training pitfall, we propose LayerNorm Scaling, which scales the variance
  of output of the layer normalization inversely by the square root of its depth.
  This simple modification mitigates the output variance explosion of deeper
  Transformer layers, improving their contribution. Our experimental results,
  spanning model sizes from 130M to 1B, demonstrate that LayerNorm Scaling
  significantly enhances LLM pre-training performance compared to Pre-LN.
  Moreover, this improvement seamlessly carries over to supervised fine-tuning.
  All these gains can be attributed to the fact that LayerNorm Scaling enables
  deeper layers to contribute more effectively during training.
paper_authors._id: 67ab189a8087b66340398b01
paper_authors.name: Wenfang Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T14:25:29.711Z
paper_authors._id: 643ce831a16fa581f3f826c9
paper_authors.avatarUrl: /avatars/2f2dffb660eee3d3c7029dd7305f5226.svg
paper_authors.fullname: Wenfang Sun
paper_authors.type: user
paper_authors.user: lmsdss
paper_authors._id: 67ab189a8087b66340398b02
paper_authors.name: Xinyuan Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:06.467Z
paper_authors._id: 6728a79b2a2e92f37afb900d
paper_authors.avatarUrl: /avatars/92106187ccbe34dfdbfc5d6d6fd63210.svg
paper_authors.fullname: XinyuanSong
paper_authors.type: user
paper_authors.user: XinyuanSong
paper_authors._id: 67ab189a8087b66340398b03
paper_authors.name: Pengxiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T09:51:15.671Z
paper_authors._id: 64245f2c089d5fae56b4549a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg
paper_authors.fullname: Pengxiang Li
paper_authors.type: user
paper_authors.user: pengxiang
paper_authors._id: 67ab189a8087b66340398b04
paper_authors.name: Lu Yin
paper_authors._id: 67ab189a8087b66340398b05
paper_authors.name: Yefeng Zheng
paper_authors._id: 67ab189a8087b66340398b06
paper_authors.name: Shiwei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T11:15:25.316Z
paper_authors._id: 65b04d2291e63920a7898c9e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b04d2291e63920a7898c9e/iUHs235G4bqK-KnH_94ti.jpeg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Shiweiliuiiiiiii
paper_title: The Curse of Depth in Large Language Models
paper_summary: In this paper, we introduce the Curse of Depth, a concept that highlights,
  explains, and addresses the recent observation in modern Large Language
  Models(LLMs) where nearly half of the layers are less effective than expected.
  We first confirm the wide existence of this phenomenon across the most popular
  families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis,
  theoretically and empirically, identifies that the underlying reason for the
  ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer
  Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer
  LLMs, its output variance exponentially grows with the model depth, which
  undesirably causes the derivative of the deep Transformer blocks to be an
  identity matrix, and therefore barely contributes to the training. To resolve
  this training pitfall, we propose LayerNorm Scaling, which scales the variance
  of output of the layer normalization inversely by the square root of its depth.
  This simple modification mitigates the output variance explosion of deeper
  Transformer layers, improving their contribution. Our experimental results,
  spanning model sizes from 130M to 1B, demonstrate that LayerNorm Scaling
  significantly enhances LLM pre-training performance compared to Pre-LN.
  Moreover, this improvement seamlessly carries over to supervised fine-tuning.
  All these gains can be attributed to the fact that LayerNorm Scaling enables
  deeper layers to contribute more effectively during training.
paper_ai_keywords: Curse of Depth Large Language Models (LLMs) Llama Mistral DeepSeek Qwen Pre-Layer Normalization (Pre-LN) Transformer LLMs Pre-LN LayerNorm Scaling Transformer blocks variance explosion identity matrix supervised fine-tuning