title: HGRN2: Gated Linear RNNs with State Expansion
summary: Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated
  competitive training speed and performance in language modeling, while offering
  efficient inference. However, the recurrent state size of HGRN remains
  relatively small, which limits its expressiveness.To address this issue,
  inspired by linear attention, we introduce a simple outer-product-based state
  expansion mechanism so that the recurrent state size can be significantly
  enlarged without introducing any additional parameters. The linear attention
  form also allows for hardware-efficient training.Our extensive experiments
  verify the advantage of HGRN2 over HGRN1 in language modeling, image
  classification, and Long Range Arena.Our largest 3B HGRN2 model slightly
  outperforms Mamba and LLaMa Architecture Transformer for language modeling in a
  controlled experiment setting; and performs competitively with many open-source
  3B models in downstream evaluation while using much fewer total training
  tokens.
paper_authors._id: 6618b29fb02a077ef1d94c23
paper_authors.name: Zhen Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:37.644Z
paper_authors._id: 642e63a53c2cf43f6d6dc5ce
paper_authors.avatarUrl: /avatars/dfd78c8d55485c22be6e616670a633e5.svg
paper_authors.fullname: zhenqin
paper_authors.type: user
paper_authors.user: Doreamonzzz
paper_authors._id: 6618b29fb02a077ef1d94c24
paper_authors.name: Songlin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:24:24.796Z
paper_authors._id: 65810ad10d738799ecbce2de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65810ad10d738799ecbce2de/e0u0Eo-4Wiahw_-cCHvIx.jpeg
paper_authors.fullname: Songlin Yang
paper_authors.type: user
paper_authors.user: sonta7
paper_authors._id: 6618b29fb02a077ef1d94c25
paper_authors.name: Weixuan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:24:46.326Z
paper_authors._id: 640c4d2d536d9fe0f000b209
paper_authors.avatarUrl: /avatars/419e59b78f7fc7f6b3ac343398d11ec1.svg
paper_authors.fullname: Weixuan Sun
paper_authors.type: user
paper_authors.user: weixuansun
paper_authors._id: 6618b29fb02a077ef1d94c26
paper_authors.name: Xuyang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:25:24.154Z
paper_authors._id: 642e4d4d6748dd4f8eeb7732
paper_authors.avatarUrl: /avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
paper_authors.fullname: Xuyang Shen
paper_authors.type: user
paper_authors.user: Ryan1122
paper_authors._id: 6618b29fb02a077ef1d94c27
paper_authors.name: Dong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:51:08.803Z
paper_authors._id: 61cd713f78c1c6b489a231fc
paper_authors.avatarUrl: /avatars/af261b85bbd10a1182372ffc459640b8.svg
paper_authors.fullname: Dong Li
paper_authors.type: user
paper_authors.user: liddalidd
paper_authors._id: 6618b29fb02a077ef1d94c28
paper_authors.name: Weigao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:25:39.776Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 6618b29fb02a077ef1d94c29
paper_authors.name: Yiran Zhong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T10:28:26.131Z
paper_authors._id: 64c525e4d68946edad6c7067
paper_authors.avatarUrl: /avatars/1b108661634af602717a4ab4b66a151f.svg
paper_authors.fullname: Yiran Zhong
paper_authors.type: user
paper_authors.user: IanZhong
paper_title: HGRN2: Gated Linear RNNs with State Expansion
paper_summary: Hierarchically gated linear RNN (HGRN,Qin et al. 2023) has demonstrated
  competitive training speed and performance in language modeling, while offering
  efficient inference. However, the recurrent state size of HGRN remains
  relatively small, which limits its expressiveness.To address this issue,
  inspired by linear attention, we introduce a simple outer-product-based state
  expansion mechanism so that the recurrent state size can be significantly
  enlarged without introducing any additional parameters. The linear attention
  form also allows for hardware-efficient training.Our extensive experiments
  verify the advantage of HGRN2 over HGRN1 in language modeling, image
  classification, and Long Range Arena.Our largest 3B HGRN2 model slightly
  outperforms Mamba and LLaMa Architecture Transformer for language modeling in a
  controlled experiment setting; and performs competitively with many open-source
  3B models in downstream evaluation while using much fewer total training
  tokens.
paper_ai_summary: The introduction of an outer-product-based state expansion mechanism in hierarchical gated linear RNN (HGRN) enhances its performance and expressiveness without additional parameters, outperforming architectures like Mamba and LLaMa in various tasks.
paper_ai_keywords: hierarchically gated linear RNN linear attention outer-product-based state expansion linear attention form hardware-efficient training Long Range Arena language modeling image classification Mamba LLaMa Architecture Transformer