title: Scaling Laws for Linear Complexity Language Models
summary: The interest in linear complexity models for large language models is on the
  rise, although their scaling capacity remains uncertain. In this study, we
  present the scaling laws for linear complexity language models to establish a
  foundation for their scalability. Specifically, we examine the scaling
  behaviors of three efficient linear architectures. These include TNL, a linear
  attention model with data-independent decay; HGRN2, a linear RNN with
  data-dependent decay; and cosFormer2, a linear attention model without decay.
  We also include LLaMA as a baseline architecture for softmax attention for
  comparison. These models were trained with six variants, ranging from 70M to 7B
  parameters on a 300B-token corpus, and evaluated with a total of 1,376
  intermediate checkpoints on various downstream tasks. These tasks include
  validation loss, commonsense reasoning, and information retrieval and
  generation. The study reveals that existing linear complexity language models
  exhibit similar scaling capabilities as conventional transformer-based models
  while also demonstrating superior linguistic proficiency and knowledge
  retention.
paper_authors._id: 667a4bbdb2401c64d7ede147
paper_authors.name: Xuyang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:02.849Z
paper_authors._id: 642e4d4d6748dd4f8eeb7732
paper_authors.avatarUrl: /avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
paper_authors.fullname: Xuyang Shen
paper_authors.type: user
paper_authors.user: Ryan1122
paper_authors._id: 667a4bbdb2401c64d7ede148
paper_authors.name: Dong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:50:57.945Z
paper_authors._id: 61cd713f78c1c6b489a231fc
paper_authors.avatarUrl: /avatars/af261b85bbd10a1182372ffc459640b8.svg
paper_authors.fullname: Dong Li
paper_authors.type: user
paper_authors.user: liddalidd
paper_authors._id: 667a4bbdb2401c64d7ede149
paper_authors.name: Ruitao Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:16:54.446Z
paper_authors._id: 65d6c5eebcf1fed7232f04d8
paper_authors.avatarUrl: /avatars/7eaebf3438323d9e6629cb25f541ffc7.svg
paper_authors.fullname: lengruitao
paper_authors.type: user
paper_authors.user: lengruitao
paper_authors._id: 667a4bbdb2401c64d7ede14a
paper_authors.name: Zhen Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:17:01.859Z
paper_authors._id: 649014b91d71e55664838d2d
paper_authors.avatarUrl: /avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
paper_authors.fullname: Zhen Qin
paper_authors.type: user
paper_authors.user: zhenqincn
paper_authors._id: 667a4bbdb2401c64d7ede14b
paper_authors.name: Weigao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:17:08.568Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 667a4bbdb2401c64d7ede14c
paper_authors.name: Yiran Zhong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T10:27:58.102Z
paper_authors._id: 64c525e4d68946edad6c7067
paper_authors.avatarUrl: /avatars/1b108661634af602717a4ab4b66a151f.svg
paper_authors.fullname: Yiran Zhong
paper_authors.type: user
paper_authors.user: IanZhong
paper_title: Scaling Laws for Linear Complexity Language Models
paper_summary: The interest in linear complexity models for large language models is on the
  rise, although their scaling capacity remains uncertain. In this study, we
  present the scaling laws for linear complexity language models to establish a
  foundation for their scalability. Specifically, we examine the scaling
  behaviors of three efficient linear architectures. These include TNL, a linear
  attention model with data-independent decay; HGRN2, a linear RNN with
  data-dependent decay; and cosFormer2, a linear attention model without decay.
  We also include LLaMA as a baseline architecture for softmax attention for
  comparison. These models were trained with six variants, ranging from 70M to 7B
  parameters on a 300B-token corpus, and evaluated with a total of 1,376
  intermediate checkpoints on various downstream tasks. These tasks include
  validation loss, commonsense reasoning, and information retrieval and
  generation. The study reveals that existing linear complexity language models
  exhibit similar scaling capabilities as conventional transformer-based models
  while also demonstrating superior linguistic proficiency and knowledge
  retention.
paper_ai_summary: Linear complexity language models show similar scalability and superior linguistic performance compared to traditional transformer-based models.
paper_ai_keywords: linear complexity models TNL linear attention model HGRN2 linear RNN cosFormer2 softmax attention validation loss commonsense reasoning information retrieval information generation