Overflow Prevention Enhances Long-Context Recurrent LLMs
A recent trend in LLMs is developing recurrent sub-quadratic models that
improve long-context processing efficiency. We investigate leading large
long-context models, focusing on how their fixed-size recurrent memory affects
their performance. Our experiments reveal that, even when these models are
trained for extended contexts, their use of long contexts remains
underutilized. Specifically, we demonstrate that a chunk-based inference
procedure, which identifies and processes only the most relevant portion of the
input can mitigate recurrent memory failures and be effective for many
long-context tasks: On LongBench, our method improves the overall performance
of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%,
RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this
simple approach also leads to state-of-the-art results in the challenging
LongBench v2 benchmark, showing competitive performance with equivalent size
Transformers. Furthermore, our findings raise questions about whether recurrent
models genuinely exploit long-range dependencies, as our single-chunk strategy
delivers stronger performance - even in tasks that presumably require
cross-context relations.
6822fa434c32644c4cad2fd7
Assaf Ben-Kish
admin_assigned
2025-05-13T08:39:13.815Z
65810ac5bc91d2ab5d8f4e03
https://cdn-avatars.huggingface.co/v1/production/uploads/65810ac5bc91d2ab5d8f4e03/mp-oYeE8vJ97YeqbKsp8N.png
Assaf Ben-Kish
user
assafbk
6822fa434c32644c4cad2fd8
Itamar Zimerman
admin_assigned
2025-05-13T08:39:19.726Z
65376feed325b3f02fb92c69
/avatars/e952918cf434d5302e9b1a404eccaf0e.svg
Itamar Zimerman
user
ItamarZ
6822fa434c32644c4cad2fd9
M. Jehanzeb Mirza
6822fa434c32644c4cad2fda
James Glass
admin_assigned
2025-05-13T08:39:38.415Z
62a793d33e101ec156cc58c4
/avatars/d3607cdb2cac1b2a3011d7441e6ea321.svg
James Glass
user
OmegaLittleBob
6822fa434c32644c4cad2fdb
Leonid Karlinsky
admin_assigned
2025-05-13T08:39:44.021Z
62558f7cebfc059817b06690
/avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
Leonid Karlinsky
user
leokarlin
6822fa434c32644c4cad2fdc
Raja Giryes
admin_assigned
2025-05-13T08:39:49.959Z
630e63148df86f1e5bf17fb6
/avatars/750dfc7f6dec2c0aa9eeb005210d3a27.svg
Raja Giryes
user
rgiryes
Overflow Prevention Enhances Long-Context Recurrent LLMs
A recent trend in LLMs is developing recurrent sub-quadratic models that
improve long-context processing efficiency. We investigate leading large
long-context models, focusing on how their fixed-size recurrent memory affects
their performance. Our experiments reveal that, even when these models are
trained for extended contexts, their use of long contexts remains
underutilized. Specifically, we demonstrate that a chunk-based inference
procedure, which identifies and processes only the most relevant portion of the
input can mitigate recurrent memory failures and be effective for many
long-context tasks: On LongBench, our method improves the overall performance
of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%,
RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this
simple approach also leads to state-of-the-art results in the challenging
LongBench v2 benchmark, showing competitive performance with equivalent size
Transformers. Furthermore, our findings raise questions about whether recurrent
models genuinely exploit long-range dependencies, as our single-chunk strategy
delivers stronger performance - even in tasks that presumably require
cross-context relations.
A chunk-based inference method improves the performance of large language models by efficiently processing long contexts, achieving state-of-the-art results in the LongBench v2 benchmark.
recurrent sub-quadratic models long-context models fixed-size recurrent memory chunk-based inference LongBench Falcon3-Mamba-Inst-7B Falcon-Mamba-Inst-7B RecurrentGemma-IT-9B RWKV6-Finch-7B long-range dependencies