Scaling Laws for Linear Complexity Language Models
The interest in linear complexity models for large language models is on the
rise, although their scaling capacity remains uncertain. In this study, we
present the scaling laws for linear complexity language models to establish a
foundation for their scalability. Specifically, we examine the scaling
behaviors of three efficient linear architectures. These include TNL, a linear
attention model with data-independent decay; HGRN2, a linear RNN with
data-dependent decay; and cosFormer2, a linear attention model without decay.
We also include LLaMA as a baseline architecture for softmax attention for
comparison. These models were trained with six variants, ranging from 70M to 7B
parameters on a 300B-token corpus, and evaluated with a total of 1,376
intermediate checkpoints on various downstream tasks. These tasks include
validation loss, commonsense reasoning, and information retrieval and
generation. The study reveals that existing linear complexity language models
exhibit similar scaling capabilities as conventional transformer-based models
while also demonstrating superior linguistic proficiency and knowledge
retention.
667a4bbdb2401c64d7ede147
Xuyang Shen
claimed_verified
2024-06-25T07:33:02.849Z
642e4d4d6748dd4f8eeb7732
/avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
Xuyang Shen
user
Ryan1122
667a4bbdb2401c64d7ede148
Dong Li
claimed_verified
2025-01-15T08:50:57.945Z
61cd713f78c1c6b489a231fc
/avatars/af261b85bbd10a1182372ffc459640b8.svg
Dong Li
user
liddalidd
667a4bbdb2401c64d7ede149
Ruitao Leng
admin_assigned
2024-06-25T08:16:54.446Z
65d6c5eebcf1fed7232f04d8
/avatars/7eaebf3438323d9e6629cb25f541ffc7.svg
lengruitao
user
lengruitao
667a4bbdb2401c64d7ede14a
Zhen Qin
admin_assigned
2024-06-25T08:17:01.859Z
649014b91d71e55664838d2d
/avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
Zhen Qin
user
zhenqincn
667a4bbdb2401c64d7ede14b
Weigao Sun
admin_assigned
2024-06-25T08:17:08.568Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
667a4bbdb2401c64d7ede14c
Yiran Zhong
claimed_verified
2025-01-17T10:27:58.102Z
64c525e4d68946edad6c7067
/avatars/1b108661634af602717a4ab4b66a151f.svg
Yiran Zhong
user
IanZhong
Scaling Laws for Linear Complexity Language Models
The interest in linear complexity models for large language models is on the
rise, although their scaling capacity remains uncertain. In this study, we
present the scaling laws for linear complexity language models to establish a
foundation for their scalability. Specifically, we examine the scaling
behaviors of three efficient linear architectures. These include TNL, a linear
attention model with data-independent decay; HGRN2, a linear RNN with
data-dependent decay; and cosFormer2, a linear attention model without decay.
We also include LLaMA as a baseline architecture for softmax attention for
comparison. These models were trained with six variants, ranging from 70M to 7B
parameters on a 300B-token corpus, and evaluated with a total of 1,376
intermediate checkpoints on various downstream tasks. These tasks include
validation loss, commonsense reasoning, and information retrieval and
generation. The study reveals that existing linear complexity language models
exhibit similar scaling capabilities as conventional transformer-based models
while also demonstrating superior linguistic proficiency and knowledge
retention.
Linear complexity language models show similar scalability and superior linguistic performance compared to traditional transformer-based models.
linear complexity models TNL linear attention model HGRN2 linear RNN cosFormer2 softmax attention validation loss commonsense reasoning information retrieval information generation