title: Can Few-shot Work in Long-Context? Recycling the Context to Generate
    Demonstrations
summary: Despite recent advancements in Large Language Models (LLMs), their
  performance on tasks involving long contexts remains sub-optimal. In-Context
  Learning (ICL) with few-shot examples may be an appealing solution to enhance
  LLM performance in this scenario; However, naively adding ICL examples with
  long context introduces challenges, including substantial token overhead added
  for each few-shot example and context mismatch between the demonstrations and
  the target query. In this work, we propose to automatically generate few-shot
  examples for long context QA tasks by recycling contexts. Specifically, given a
  long input context (1-3k tokens) and a query, we generate additional
  query-output pairs from the given context as few-shot examples, while
  introducing the context only once. This ensures that the demonstrations are
  leveraging the same context as the target query while only adding a small
  number of tokens to the prompt. We further enhance each demonstration by
  instructing the model to explicitly identify the relevant paragraphs before the
  answer, which improves performance while providing fine-grained attribution to
  the answer source. We apply our method on multiple LLMs and obtain substantial
  improvements (+23\% on average across models) on various QA datasets with long
  context, especially when the answer lies within the middle of the context.
  Surprisingly, despite introducing only single-hop ICL examples, LLMs also
  successfully generalize to multi-hop long-context QA using our approach.
paper_authors._id: 667afc55835224aefa0464db
paper_authors.name: Arie Cattan
paper_authors._id: 667afc55835224aefa0464dc
paper_authors.name: Alon Jacovi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:42:20.464Z
paper_authors._id: 6596c90dfe0f9826a63654d0
paper_authors.avatarUrl: /avatars/e616fbdff1a345ad22081f5cd019329a.svg
paper_authors.fullname: Alon Jacovi
paper_authors.type: user
paper_authors.user: alonjacovi
paper_authors._id: 667afc55835224aefa0464dd
paper_authors.name: Alex Fabrikant
paper_authors._id: 667afc55835224aefa0464de
paper_authors.name: Jonathan Herzig
paper_authors._id: 667afc55835224aefa0464df
paper_authors.name: Roee Aharoni
paper_authors._id: 667afc55835224aefa0464e0
paper_authors.name: Hannah Rashkin
paper_authors._id: 667afc55835224aefa0464e1
paper_authors.name: Dror Marcus
paper_authors._id: 667afc55835224aefa0464e2
paper_authors.name: Avinatan Hassidim
paper_authors._id: 667afc55835224aefa0464e3
paper_authors.name: Yossi Matias
paper_authors._id: 667afc55835224aefa0464e4
paper_authors.name: Idan Szpektor
paper_authors._id: 667afc55835224aefa0464e5
paper_authors.name: Avi Caciularu
paper_title: Can Few-shot Work in Long-Context? Recycling the Context to Generate
    Demonstrations
paper_summary: Despite recent advancements in Large Language Models (LLMs), their
  performance on tasks involving long contexts remains sub-optimal. In-Context
  Learning (ICL) with few-shot examples may be an appealing solution to enhance
  LLM performance in this scenario; However, naively adding ICL examples with
  long context introduces challenges, including substantial token overhead added
  for each few-shot example and context mismatch between the demonstrations and
  the target query. In this work, we propose to automatically generate few-shot
  examples for long context QA tasks by recycling contexts. Specifically, given a
  long input context (1-3k tokens) and a query, we generate additional
  query-output pairs from the given context as few-shot examples, while
  introducing the context only once. This ensures that the demonstrations are
  leveraging the same context as the target query while only adding a small
  number of tokens to the prompt. We further enhance each demonstration by
  instructing the model to explicitly identify the relevant paragraphs before the
  answer, which improves performance while providing fine-grained attribution to
  the answer source. We apply our method on multiple LLMs and obtain substantial
  improvements (+23\% on average across models) on various QA datasets with long
  context, especially when the answer lies within the middle of the context.
  Surprisingly, despite introducing only single-hop ICL examples, LLMs also
  successfully generalize to multi-hop long-context QA using our approach.
paper_ai_keywords: Large Language Models (LLMs) In-Context Learning (ICL) long contexts token overhead context mismatch modular few-shot examples long input context query-output pairs self-adaptive relevant paragraphs QA tasks multi-hop long-context QA