HGRN2: Gated Linear RNNs with State Expansion
Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated
competitive training speed and performance in language modeling, while offering
efficient inference. However, the recurrent state size of HGRN remains
relatively small, which limits its expressiveness.To address this issue,
inspired by linear attention, we introduce a simple outer-product-based state
expansion mechanism so that the recurrent state size can be significantly
enlarged without introducing any additional parameters. The linear attention
form also allows for hardware-efficient training.Our extensive experiments
verify the advantage of HGRN2 over HGRN1 in language modeling, image
classification, and Long Range Arena.Our largest 3B HGRN2 model slightly
outperforms Mamba and LLaMa Architecture Transformer for language modeling in a
controlled experiment setting; and performs competitively with many open-source
3B models in downstream evaluation while using much fewer total training
tokens.
6618b29fb02a077ef1d94c23
Zhen Qin
claimed_verified
2024-12-04T09:47:37.644Z
642e63a53c2cf43f6d6dc5ce
/avatars/dfd78c8d55485c22be6e616670a633e5.svg
zhenqin
user
Doreamonzzz
6618b29fb02a077ef1d94c24
Songlin Yang
admin_assigned
2024-04-18T12:24:24.796Z
65810ad10d738799ecbce2de
https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
Songlin Yang
user
sonta7
6618b29fb02a077ef1d94c25
Weixuan Sun
admin_assigned
2024-04-18T12:24:46.326Z
640c4d2d536d9fe0f000b209
/avatars/419e59b78f7fc7f6b3ac343398d11ec1.svg
Weixuan Sun
user
weixuansun
6618b29fb02a077ef1d94c26
Xuyang Shen
admin_assigned
2024-04-18T12:25:24.154Z
642e4d4d6748dd4f8eeb7732
/avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
Xuyang Shen
user
Ryan1122
6618b29fb02a077ef1d94c27
Dong Li
claimed_verified
2025-01-15T08:51:08.803Z
61cd713f78c1c6b489a231fc
/avatars/af261b85bbd10a1182372ffc459640b8.svg
Dong Li
user
liddalidd
6618b29fb02a077ef1d94c28
Weigao Sun
admin_assigned
2024-04-18T12:25:39.776Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
6618b29fb02a077ef1d94c29
Yiran Zhong
claimed_verified
2025-01-17T10:28:26.131Z
64c525e4d68946edad6c7067
/avatars/1b108661634af602717a4ab4b66a151f.svg
Yiran Zhong
user
IanZhong
HGRN2: Gated Linear RNNs with State Expansion
Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated
competitive training speed and performance in language modeling, while offering
efficient inference. However, the recurrent state size of HGRN remains
relatively small, which limits its expressiveness.To address this issue,
inspired by linear attention, we introduce a simple outer-product-based state
expansion mechanism so that the recurrent state size can be significantly
enlarged without introducing any additional parameters. The linear attention
form also allows for hardware-efficient training.Our extensive experiments
verify the advantage of HGRN2 over HGRN1 in language modeling, image
classification, and Long Range Arena.Our largest 3B HGRN2 model slightly
outperforms Mamba and LLaMa Architecture Transformer for language modeling in a
controlled experiment setting; and performs competitively with many open-source
3B models in downstream evaluation while using much fewer total training
tokens.
The introduction of an outer-product-based state expansion mechanism in hierarchical gated linear RNN (HGRN) enhances its performance and expressiveness without additional parameters, outperforming architectures like Mamba and LLaMa in various tasks.
hierarchically gated linear RNN linear attention outer-product-based state expansion linear attention form hardware-efficient training Long Range Arena language modeling image classification Mamba LLaMa Architecture Transformer