title: Overflow Prevention Enhances Long-Context Recurrent LLMs
summary: A recent trend in LLMs is developing recurrent sub-quadratic models that
  improve long-context processing efficiency. We investigate leading large
  long-context models, focusing on how their fixed-size recurrent memory affects
  their performance. Our experiments reveal that, even when these models are
  trained for extended contexts, their use of long contexts remains
  underutilized. Specifically, we demonstrate that a chunk-based inference
  procedure, which identifies and processes only the most relevant portion of the
  input can mitigate recurrent memory failures and be effective for many
  long-context tasks: On LongBench, our method improves the overall performance
  of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%,
  RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this
  simple approach also leads to state-of-the-art results in the challenging
  LongBench v2 benchmark, showing competitive performance with equivalent size
  Transformers. Furthermore, our findings raise questions about whether recurrent
  models genuinely exploit long-range dependencies, as our single-chunk strategy
  delivers stronger performance - even in tasks that presumably require
  cross-context relations.
paper_authors._id: 6822fa434c32644c4cad2fd7
paper_authors.name: Assaf Ben-Kish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:39:13.815Z
paper_authors._id: 65810ac5bc91d2ab5d8f4e03
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65810ac5bc91d2ab5d8f4e03/mp-oYeE8vJ97YeqbKsp8N.png
paper_authors.fullname: Assaf Ben-Kish
paper_authors.type: user
paper_authors.user: assafbk
paper_authors._id: 6822fa434c32644c4cad2fd8
paper_authors.name: Itamar Zimerman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:39:19.726Z
paper_authors._id: 65376feed325b3f02fb92c69
paper_authors.avatarUrl: /avatars/e952918cf434d5302e9b1a404eccaf0e.svg
paper_authors.fullname: Itamar Zimerman
paper_authors.type: user
paper_authors.user: ItamarZ
paper_authors._id: 6822fa434c32644c4cad2fd9
paper_authors.name: M. Jehanzeb Mirza
paper_authors._id: 6822fa434c32644c4cad2fda
paper_authors.name: James Glass
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:39:38.415Z
paper_authors._id: 62a793d33e101ec156cc58c4
paper_authors.avatarUrl: /avatars/d3607cdb2cac1b2a3011d7441e6ea321.svg
paper_authors.fullname: James Glass
paper_authors.type: user
paper_authors.user: OmegaLittleBob
paper_authors._id: 6822fa434c32644c4cad2fdb
paper_authors.name: Leonid Karlinsky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:39:44.021Z
paper_authors._id: 62558f7cebfc059817b06690
paper_authors.avatarUrl: /avatars/40fbfdbeb0b33e21a48478deb2a742d1.svg
paper_authors.fullname: Leonid Karlinsky
paper_authors.type: user
paper_authors.user: leokarlin
paper_authors._id: 6822fa434c32644c4cad2fdc
paper_authors.name: Raja Giryes
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:39:49.959Z
paper_authors._id: 630e63148df86f1e5bf17fb6
paper_authors.avatarUrl: /avatars/750dfc7f6dec2c0aa9eeb005210d3a27.svg
paper_authors.fullname: Raja Giryes
paper_authors.type: user
paper_authors.user: rgiryes
paper_title: Overflow Prevention Enhances Long-Context Recurrent LLMs
paper_summary: A recent trend in LLMs is developing recurrent sub-quadratic models that
  improve long-context processing efficiency. We investigate leading large
  long-context models, focusing on how their fixed-size recurrent memory affects
  their performance. Our experiments reveal that, even when these models are
  trained for extended contexts, their use of long contexts remains
  underutilized. Specifically, we demonstrate that a chunk-based inference
  procedure, which identifies and processes only the most relevant portion of the
  input can mitigate recurrent memory failures and be effective for many
  long-context tasks: On LongBench, our method improves the overall performance
  of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%,
  RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this
  simple approach also leads to state-of-the-art results in the challenging
  LongBench v2 benchmark, showing competitive performance with equivalent size
  Transformers. Furthermore, our findings raise questions about whether recurrent
  models genuinely exploit long-range dependencies, as our single-chunk strategy
  delivers stronger performance - even in tasks that presumably require
  cross-context relations.
paper_ai_summary: A chunk-based inference method improves the performance of large language models by efficiently processing long contexts, achieving state-of-the-art results in the LongBench v2 benchmark.
paper_ai_keywords: recurrent sub-quadratic models long-context models fixed-size recurrent memory chunk-based inference LongBench Falcon3-Mamba-Inst-7B Falcon-Mamba-Inst-7B RecurrentGemma-IT-9B RWKV6-Finch-7B long-range dependencies