Is It Really Long Context if All You Need Is Retrieval? Towards
  Genuinely Difficult Long Context NLP
Improvements in language models' capabilities have pushed their applications
towards longer contexts, making long-context evaluation and development an
active research area. However, many disparate use-cases are grouped together
under the umbrella term of "long-context", defined simply by the total length
of the model's input, including - for example - Needle-in-a-Haystack tasks,
book summarization, and information aggregation. Given their varied difficulty,
in this position paper we argue that conflating different tasks by their
context length is unproductive. As a community, we require a more precise
vocabulary to understand what makes long-context tasks similar or different. We
propose to unpack the taxonomy of long-context based on the properties that
make them more difficult with longer contexts. We propose two orthogonal axes
of difficulty: (I) Diffusion: How hard is it to find the necessary information
in the context? (II) Scope: How much necessary information is there to find? We
survey the literature on long-context, provide justification for this taxonomy
as an informative descriptor, and situate the literature with respect to it. We
conclude that the most difficult and interesting settings, whose necessary
information is very long and highly diffused within the input, is severely
under-explored. By using a descriptive vocabulary and discussing the relevant
properties of difficulty in long-context, we can implement more informed
research in this area. We call for a careful design of tasks and benchmarks
with distinctly long context, taking into account the characteristics that make
it qualitatively different from shorter context.
66840e1922572e72982cf73b
Omer Goldman
extracted_confirmed
2024-07-02T15:16:01.407Z
668418b8e2e0c3b2150efeed
/avatars/8eaa0f3d112e5850e2ae383f269c5734.svg
omer goldman
user
omergoldman
66840e1922572e72982cf73c
Alon Jacovi
claimed_verified
2024-08-08T07:42:22.088Z
6596c90dfe0f9826a63654d0
/avatars/e616fbdff1a345ad22081f5cd019329a.svg
Alon Jacovi
user
alonjacovi
66840e1922572e72982cf73d
Aviv Slobodkin
claimed_verified
2024-07-02T21:08:54.060Z
631da07f6d6a5870f3d2c375
/avatars/242e344dca08057bdf1eef09f69b41b2.svg
Aviv Slobodkin
user
lovodkin93
66840e1922572e72982cf73e
Aviya Maimon
66840e1922572e72982cf73f
Ido Dagan
66840e1922572e72982cf740
Reut Tsarfaty
Is It Really Long Context if All You Need Is Retrieval? Towards
  Genuinely Difficult Long Context NLP
Improvements in language models' capabilities have pushed their applications
towards longer contexts, making long-context evaluation and development an
active research area. However, many disparate use-cases are grouped together
under the umbrella term of "long-context", defined simply by the total length
of the model's input, including - for example - Needle-in-a-Haystack tasks,
book summarization, and information aggregation. Given their varied difficulty,
in this position paper we argue that conflating different tasks by their
context length is unproductive. As a community, we require a more precise
vocabulary to understand what makes long-context tasks similar or different. We
propose to unpack the taxonomy of long-context based on the properties that
make them more difficult with longer contexts. We propose two orthogonal axes
of difficulty: (I) Diffusion: How hard is it to find the necessary information
in the context? (II) Scope: How much necessary information is there to find? We
survey the literature on long-context, provide justification for this taxonomy
as an informative descriptor, and situate the literature with respect to it. We
conclude that the most difficult and interesting settings, whose necessary
information is very long and highly diffused within the input, is severely
under-explored. By using a descriptive vocabulary and discussing the relevant
properties of difficulty in long-context, we can implement more informed
research in this area. We call for a careful design of tasks and benchmarks
with distinctly long context, taking into account the characteristics that make
it qualitatively different from shorter context.
A position paper argues for a more precise taxonomy of long-context tasks based on their diffusion and scope of necessary information, noting that highly diffused and lengthy information in long contexts is under-explored.
diffusion scope long-context tasks taxonomy