Lizard: An Efficient Linearization Framework for Large Language Models
We propose Lizard, a linearization framework that transforms pretrained
Transformer-based Large Language Models (LLMs) into flexible, subquadratic
architectures for infinite-context generation. Transformer-based LLMs face
significant memory and computational bottlenecks as context lengths increase,
due to the quadratic complexity of softmax attention and the growing key-value
(KV) cache. Lizard addresses these limitations by introducing a subquadratic
attention mechanism that closely approximates softmax attention while
preserving the output quality. Unlike previous linearization methods, which are
often limited by fixed model structures and therefore exclude gating
mechanisms, Lizard incorporates a gating module inspired by recent
state-of-the-art linear models. This enables adaptive memory control, supports
constant-memory inference, offers strong length generalization, and allows more
flexible model design. Lizard combines gated linear attention for global
context compression with sliding window attention enhanced by meta memory,
forming a hybrid mechanism that captures both long-range dependencies and
fine-grained local interactions. Moreover, we introduce a hardware-aware
algorithm that accelerates the training speed of our models. Extensive
experiments show that Lizard achieves near-lossless recovery of the teacher
model's performance across standard language modeling tasks, while
significantly outperforming previous linearization methods. On the 5-shot MMLU
benchmark, Lizard improves over prior models by 18 points and shows significant
improvements on associative recall tasks.
68786e45001546c83aa4f9a0
Chien Van Nguyen
68786e45001546c83aa4f9a1
Ruiyi Zhang
68786e45001546c83aa4f9a2
Hanieh Deilamsalehy
admin_assigned
2025-07-17T09:36:52.637Z
652767bfbdcf00b9b9ac9a74
/avatars/2cc8e9167562f364f0c25410f13a9d62.svg
Hanieh Deilamsalehy
user
haniehds
68786e45001546c83aa4f9a3
Puneet Mathur
68786e45001546c83aa4f9a4
Viet Dac Lai
68786e45001546c83aa4f9a5
Haoliang Wang
68786e45001546c83aa4f9a6
Jayakumar Subramanian
admin_assigned
2025-07-17T09:37:03.196Z
66d32a678819c81cce2052f4
/avatars/3a5b40ef9e9e73512743756d1c24ab6c.svg
Jayakumar Subramanian
user
jasubram
68786e45001546c83aa4f9a7
Ryan A. Rossi
68786e45001546c83aa4f9a8
Trung Bui
admin_assigned
2025-07-17T09:37:13.349Z
67f1035155fe17a33dc71f23
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ohlRrzUI8VwvyYYxFXJuY.png
Trung Bui
user
TrungBui1111
68786e45001546c83aa4f9a9
Nikos Vlassis
admin_assigned
2025-07-17T09:36:17.107Z
675346f0ab1d47a36ca60b89
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mqWUUxAuFn3DsgIqoF_ah.png
Nikos Vlassis
user
Nikosapa
68786e45001546c83aa4f9aa
Franck Dernoncourt
claimed_verified
2025-07-17T08:14:23.916Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
68786e45001546c83aa4f9ab
Thien Huu Nguyen
admin_assigned
2025-07-17T09:36:11.546Z
64804fad8c6a3b8f11f73912
/avatars/61e37a91d4bba35fda9bf52aadd87745.svg
Thien Huu Nguyen
user
anoperson
Lizard: An Efficient Linearization Framework for Large Language Models
We propose Lizard, a linearization framework that transforms pretrained
Transformer-based Large Language Models (LLMs) into flexible, subquadratic
architectures for infinite-context generation. Transformer-based LLMs face
significant memory and computational bottlenecks as context lengths increase,
due to the quadratic complexity of softmax attention and the growing key-value
(KV) cache. Lizard addresses these limitations by introducing a subquadratic
attention mechanism that closely approximates softmax attention while
preserving the output quality. Unlike previous linearization methods, which are
often limited by fixed model structures and therefore exclude gating
mechanisms, Lizard incorporates a gating module inspired by recent
state-of-the-art linear models. This enables adaptive memory control, supports
constant-memory inference, offers strong length generalization, and allows more
flexible model design. Lizard combines gated linear attention for global
context compression with sliding window attention enhanced by meta memory,
forming a hybrid mechanism that captures both long-range dependencies and
fine-grained local interactions. Moreover, we introduce a hardware-aware
algorithm that accelerates the training speed of our models. Extensive
experiments show that Lizard achieves near-lossless recovery of the teacher
model's performance across standard language modeling tasks, while
significantly outperforming previous linearization methods. On the 5-shot MMLU
benchmark, Lizard improves over prior models by 18 points and shows significant
improvements on associative recall tasks.
Lizard is a linearization framework that transforms Transformer-based LLMs into subquadratic architectures for efficient infinite-context generation, using a hybrid attention mechanism and hardware-aware training.
Transformer-based LLMs subquadratic architectures softmax attention key-value (KV) cache subquadratic attention mechanism gating module adaptive memory control constant-memory inference length generalization gated linear attention sliding window attention meta memory hardware-aware algorithm MMLU benchmark associative recall tasks