title: NoLiMa: Long-Context Evaluation Beyond Literal Matching
summary: Recent large language models (LLMs) support long contexts ranging from 128K
  to 1M tokens. A popular method for evaluating these capabilities is the
  needle-in-a-haystack (NIAH) test, which involves retrieving a "needle"
  (relevant information) from a "haystack" (long irrelevant context). Extensions
  of this approach include increasing distractors, fact chaining, and in-context
  reasoning. However, in these benchmarks, models can exploit existing literal
  matches between the needle and haystack to simplify the task. To address this,
  we introduce NoLiMa, a benchmark extending NIAH with a carefully designed
  needle set, where questions and needles have minimal lexical overlap, requiring
  models to infer latent associations to locate the needle within the haystack.
  We evaluate 12 popular LLMs that claim to support contexts of at least 128K
  tokens. While they perform well in short contexts (<1K), performance degrades
  significantly as context length increases. At 32K, for instance, 10 models drop
  below 50% of their strong short-length baselines. Even GPT-4o, one of the
  top-performing exceptions, experiences a reduction from an almost-perfect
  baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the
  increased difficulty the attention mechanism faces in longer contexts when
  literal matches are absent, making it harder to retrieve relevant information.
paper_authors._id: 67aa583c3a878652daeae02e
paper_authors.name: Ali Modarressi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:42.560Z
paper_authors._id: 60e4738a8c0ddd18fc27ff88
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e4738a8c0ddd18fc27ff88/lpLeeIW8r85RTY4fGZTva.jpeg
paper_authors.fullname: Ali Modarressi
paper_authors.type: user
paper_authors.user: amodaresi
paper_authors._id: 67aa583c3a878652daeae02f
paper_authors.name: Hanieh Deilamsalehy
paper_authors._id: 67aa583c3a878652daeae030
paper_authors.name: Franck Dernoncourt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:26:01.327Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 67aa583c3a878652daeae031
paper_authors.name: Trung Bui
paper_authors._id: 67aa583c3a878652daeae032
paper_authors.name: Ryan A. Rossi
paper_authors._id: 67aa583c3a878652daeae033
paper_authors.name: Seunghyun Yoon
paper_authors._id: 67aa583c3a878652daeae034
paper_authors.name: Hinrich Schütze
paper_title: NoLiMa: Long-Context Evaluation Beyond Literal Matching
paper_summary: Recent large language models (LLMs) support long contexts ranging from 128K
  to 1M tokens. A popular method for evaluating these capabilities is the
  needle-in-a-haystack (NIAH) test, which involves retrieving a "needle"
  (relevant information) from a "haystack" (long irrelevant context). Extensions
  of this approach include increasing distractors, fact chaining, and in-context
  reasoning. However, in these benchmarks, models can exploit existing literal
  matches between the needle and haystack to simplify the task. To address this,
  we introduce NoLiMa, a benchmark extending NIAH with a carefully designed
  needle set, where questions and needles have minimal lexical overlap, requiring
  models to infer latent associations to locate the needle within the haystack.
  We evaluate 12 popular LLMs that claim to support contexts of at least 128K
  tokens. While they perform well in short contexts (<1K), performance degrades
  significantly as context length increases. At 32K, for instance, 10 models drop
  below 50% of their strong short-length baselines. Even GPT-4o, one of the
  top-performing exceptions, experiences a reduction from an almost-perfect
  baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the
  increased difficulty the attention mechanism faces in longer contexts when
  literal matches are absent, making it harder to retrieve relevant information.
paper_ai_summary: NoLiMa benchmark tests large language models' ability to retrieve relevant information from long contexts without literal matches, revealing performance degradation as context length increases.
paper_ai_keywords: large language models needle-in-a-haystack test distractors fact chaining in-context reasoning attention mechanism