title: Revisiting In-Context Learning with Long Context Language Models
summary: In-Context Learning (ICL) is a technique by which language models make
  predictions based on examples provided in their input context. Previously,
  their context window size imposed a limit on the number of examples that can be
  shown, making example selection techniques crucial for identifying the
  maximally effective set of examples. However, the recent advent of Long Context
  Language Models (LCLMs) has significantly increased the number of examples that
  can be included in context, raising an important question of whether ICL
  performance in a many-shot regime is still sensitive to the method of sample
  selection. To answer this, we revisit these approaches in the context of LCLMs
  through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we
  observe that sophisticated example selection techniques do not yield
  significant improvements over a simple random sample selection method. Instead,
  we find that the advent of LCLMs has fundamentally shifted the challenge of ICL
  from that of selecting the most effective examples to that of collecting
  sufficient examples to fill the context window. Specifically, in certain
  datasets, including all available examples does not fully utilize the context
  window; however, by augmenting the examples in context with a simple data
  augmentation approach, we substantially improve ICL performance by 5%.
paper_authors._id: 676a211eb16181133547681b
paper_authors.name: Jinheon Baek
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:41.247Z
paper_authors._id: 63036b6c5c70c21d0ea79d48
paper_authors.avatarUrl: /avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
paper_authors.fullname: Jinheon Baek
paper_authors.type: user
paper_authors.user: jinheon
paper_authors._id: 676a211eb16181133547681c
paper_authors.name: Sun Jae Lee
paper_authors._id: 676a211eb16181133547681d
paper_authors.name: Prakhar Gupta
paper_authors._id: 676a211eb16181133547681e
paper_authors.name: Geunseob
paper_authors._id: 676a211eb16181133547681f
paper_authors.name: Oh
paper_authors._id: 676a211eb161811335476820
paper_authors.name: Siddharth Dalmia
paper_authors._id: 676a211eb161811335476821
paper_authors.name: Prateek Kolhar
paper_title: Revisiting In-Context Learning with Long Context Language Models
paper_summary: In-Context Learning (ICL) is a technique by which language models make
  predictions based on examples provided in their input context. Previously,
  their context window size imposed a limit on the number of examples that can be
  shown, making example selection techniques crucial for identifying the
  maximally effective set of examples. However, the recent advent of Long Context
  Language Models (LCLMs) has significantly increased the number of examples that
  can be included in context, raising an important question of whether ICL
  performance in a many-shot regime is still sensitive to the method of sample
  selection. To answer this, we revisit these approaches in the context of LCLMs
  through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we
  observe that sophisticated example selection techniques do not yield
  significant improvements over a simple random sample selection method. Instead,
  we find that the advent of LCLMs has fundamentally shifted the challenge of ICL
  from that of selecting the most effective examples to that of collecting
  sufficient examples to fill the context window. Specifically, in certain
  datasets, including all available examples does not fully utilize the context
  window; however, by augmenting the examples in context with a simple data
  augmentation approach, we substantially improve ICL performance by 5%.
paper_ai_summary: Long Context Language Models reduce the sensitivity of In-Context Learning to example selection techniques and instead emphasize the importance of sufficient and augmented examples.
paper_ai_keywords: In-Context Learning Long Context Language Models