title: Is It Really Long Context if All You Need Is Retrieval? Towards
    Genuinely Difficult Long Context NLP
summary: Improvements in language models' capabilities have pushed their applications
  towards longer contexts, making long-context evaluation and development an
  active research area. However, many disparate use-cases are grouped together
  under the umbrella term of "long-context", defined simply by the total length
  of the model's input, including - for example - Needle-in-a-Haystack tasks,
  book summarization, and information aggregation. Given their varied difficulty,
  in this position paper we argue that conflating different tasks by their
  context length is unproductive. As a community, we require a more precise
  vocabulary to understand what makes long-context tasks similar or different. We
  propose to unpack the taxonomy of long-context based on the properties that
  make them more difficult with longer contexts. We propose two orthogonal axes
  of difficulty: (I) Diffusion: How hard is it to find the necessary information
  in the context? (II) Scope: How much necessary information is there to find? We
  survey the literature on long-context, provide justification for this taxonomy
  as an informative descriptor, and situate the literature with respect to it. We
  conclude that the most difficult and interesting settings, whose necessary
  information is very long and highly diffused within the input, is severely
  under-explored. By using a descriptive vocabulary and discussing the relevant
  properties of difficulty in long-context, we can implement more informed
  research in this area. We call for a careful design of tasks and benchmarks
  with distinctly long context, taking into account the characteristics that make
  it qualitatively different from shorter context.
paper_authors._id: 66840e1922572e72982cf73b
paper_authors.name: Omer Goldman
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-02T15:16:01.407Z
paper_authors._id: 668418b8e2e0c3b2150efeed
paper_authors.avatarUrl: /avatars/8eaa0f3d112e5850e2ae383f269c5734.svg
paper_authors.fullname: omer goldman
paper_authors.type: user
paper_authors.user: omergoldman
paper_authors._id: 66840e1922572e72982cf73c
paper_authors.name: Alon Jacovi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:42:22.088Z
paper_authors._id: 6596c90dfe0f9826a63654d0
paper_authors.avatarUrl: /avatars/e616fbdff1a345ad22081f5cd019329a.svg
paper_authors.fullname: Alon Jacovi
paper_authors.type: user
paper_authors.user: alonjacovi
paper_authors._id: 66840e1922572e72982cf73d
paper_authors.name: Aviv Slobodkin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T21:08:54.060Z
paper_authors._id: 631da07f6d6a5870f3d2c375
paper_authors.avatarUrl: /avatars/242e344dca08057bdf1eef09f69b41b2.svg
paper_authors.fullname: Aviv Slobodkin
paper_authors.type: user
paper_authors.user: lovodkin93
paper_authors._id: 66840e1922572e72982cf73e
paper_authors.name: Aviya Maimon
paper_authors._id: 66840e1922572e72982cf73f
paper_authors.name: Ido Dagan
paper_authors._id: 66840e1922572e72982cf740
paper_authors.name: Reut Tsarfaty
paper_title: Is It Really Long Context if All You Need Is Retrieval? Towards
    Genuinely Difficult Long Context NLP
paper_summary: Improvements in language models' capabilities have pushed their applications
  towards longer contexts, making long-context evaluation and development an
  active research area. However, many disparate use-cases are grouped together
  under the umbrella term of "long-context", defined simply by the total length
  of the model's input, including - for example - Needle-in-a-Haystack tasks,
  book summarization, and information aggregation. Given their varied difficulty,
  in this position paper we argue that conflating different tasks by their
  context length is unproductive. As a community, we require a more precise
  vocabulary to understand what makes long-context tasks similar or different. We
  propose to unpack the taxonomy of long-context based on the properties that
  make them more difficult with longer contexts. We propose two orthogonal axes
  of difficulty: (I) Diffusion: How hard is it to find the necessary information
  in the context? (II) Scope: How much necessary information is there to find? We
  survey the literature on long-context, provide justification for this taxonomy
  as an informative descriptor, and situate the literature with respect to it. We
  conclude that the most difficult and interesting settings, whose necessary
  information is very long and highly diffused within the input, is severely
  under-explored. By using a descriptive vocabulary and discussing the relevant
  properties of difficulty in long-context, we can implement more informed
  research in this area. We call for a careful design of tasks and benchmarks
  with distinctly long context, taking into account the characteristics that make
  it qualitatively different from shorter context.