title: Liger: Linearizing Large Language Models to Gated Recurrent Structures
summary: Transformers with linear recurrent modeling offer linear-time training and
  constant-memory inference. Despite their demonstrated efficiency and
  performance, pretraining such non-standard architectures from scratch remains
  costly and risky. The linearization of large language models (LLMs) transforms
  pretrained standard models into linear recurrent structures, enabling more
  efficient deployment. However, current linearization methods typically
  introduce additional feature map modules that require extensive fine-tuning and
  overlook the gating mechanisms used in state-of-the-art linear recurrent
  models. To address these issues, this paper presents Liger, short for
  Linearizing LLMs to gated recurrent structures. Liger is a novel approach for
  converting pretrained LLMs into gated linear recurrent models without adding
  extra parameters. It repurposes the pretrained key matrix weights to construct
  diverse gating mechanisms, facilitating the formation of various gated
  recurrent structures while avoiding the need to train additional components
  from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA),
  Liger restores the performance of the linearized gated recurrent models to
  match that of the original LLMs. Additionally, we introduce Liger Attention, an
  intra-layer hybrid attention mechanism, which significantly recovers 93\% of
  the Transformer-based LLM at 0.02\% pre-training tokens during the
  linearization process, achieving competitive results across multiple
  benchmarks, as validated on models ranging from 1B to 8B parameters. Code is
  available at https://github.com/OpenSparseLLMs/Linearization.
paper_authors._id: 67c6b05f35198d0f397adc98
paper_authors.name: Disen Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:34:46.117Z
paper_authors._id: 66ea643899af9ac3463639b1
paper_authors.avatarUrl: /avatars/252d470e761a57834dee3dbc60dfefed.svg
paper_authors.fullname: Disen Lan
paper_authors.type: user
paper_authors.user: landisen
paper_authors._id: 67c6b05f35198d0f397adc99
paper_authors.name: Weigao Sun
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-04T08:10:52.130Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 67c6b05f35198d0f397adc9a
paper_authors.name: Jiaxi Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:04:18.982Z
paper_authors._id: 665dc35752ff9daa9ba5a4ed
paper_authors.avatarUrl: /avatars/df8b01879d97e599b610fa51414d3a18.svg
paper_authors.fullname: Hu Jiaxi
paper_authors.type: user
paper_authors.user: Jiaxihu2
paper_authors._id: 67c6b05f35198d0f397adc9b
paper_authors.name: Jusen Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:04:26.432Z
paper_authors._id: 65003e857804f04a163328d9
paper_authors.avatarUrl: /avatars/fe32150aabfde8d283b38ccebcf6982e.svg
paper_authors.fullname: Jusen Du
paper_authors.type: user
paper_authors.user: JusenK
paper_authors._id: 67c6b05f35198d0f397adc9c
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:52:49.648Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_title: Liger: Linearizing Large Language Models to Gated Recurrent Structures
paper_summary: Transformers with linear recurrent modeling offer linear-time training and
  constant-memory inference. Despite their demonstrated efficiency and
  performance, pretraining such non-standard architectures from scratch remains
  costly and risky. The linearization of large language models (LLMs) transforms
  pretrained standard models into linear recurrent structures, enabling more
  efficient deployment. However, current linearization methods typically
  introduce additional feature map modules that require extensive fine-tuning and
  overlook the gating mechanisms used in state-of-the-art linear recurrent
  models. To address these issues, this paper presents Liger, short for
  Linearizing LLMs to gated recurrent structures. Liger is a novel approach for
  converting pretrained LLMs into gated linear recurrent models without adding
  extra parameters. It repurposes the pretrained key matrix weights to construct
  diverse gating mechanisms, facilitating the formation of various gated
  recurrent structures while avoiding the need to train additional components
  from scratch. Using lightweight fine-tuning with Low-Rank Adaptation (LoRA),
  Liger restores the performance of the linearized gated recurrent models to
  match that of the original LLMs. Additionally, we introduce Liger Attention, an
  intra-layer hybrid attention mechanism, which significantly recovers 93\% of
  the Transformer-based LLM at 0.02\% pre-training tokens during the
  linearization process, achieving competitive results across multiple
  benchmarks, as validated on models ranging from 1B to 8B parameters. Code is
  available at https://github.com/OpenSparseLLMs/Linearization.
paper_ai_summary: Liger, a novel method, converts pretrained large language models into gated linear recurrent structures using pretrained weights and lightweight fine-tuning, recovering performance and improving efficiency.
paper_ai_keywords: transformers linear recurrent modeling linearization large language models LLMs gated recurrent models Liger key matrix weights gating mechanisms Low-Rank Adaptation LoRA Liger Attention intra-layer hybrid attention mechanism