Revisiting In-Context Learning with Long Context Language Models
In-Context Learning (ICL) is a technique by which language models make
predictions based on examples provided in their input context. Previously,
their context window size imposed a limit on the number of examples that can be
shown, making example selection techniques crucial for identifying the
maximally effective set of examples. However, the recent advent of Long Context
Language Models (LCLMs) has significantly increased the number of examples that
can be included in context, raising an important question of whether ICL
performance in a many-shot regime is still sensitive to the method of sample
selection. To answer this, we revisit these approaches in the context of LCLMs
through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we
observe that sophisticated example selection techniques do not yield
significant improvements over a simple random sample selection method. Instead,
we find that the advent of LCLMs has fundamentally shifted the challenge of ICL
from that of selecting the most effective examples to that of collecting
sufficient examples to fill the context window. Specifically, in certain
datasets, including all available examples does not fully utilize the context
window; however, by augmenting the examples in context with a simple data
augmentation approach, we substantially improve ICL performance by 5%.
676a211eb16181133547681b
Jinheon Baek
claimed_verified
2024-12-30T19:35:41.247Z
63036b6c5c70c21d0ea79d48
/avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
Jinheon Baek
user
jinheon
676a211eb16181133547681c
Sun Jae Lee
676a211eb16181133547681d
Prakhar Gupta
676a211eb16181133547681e
Geunseob
676a211eb16181133547681f
Oh
676a211eb161811335476820
Siddharth Dalmia
676a211eb161811335476821
Prateek Kolhar
Revisiting In-Context Learning with Long Context Language Models
In-Context Learning (ICL) is a technique by which language models make
predictions based on examples provided in their input context. Previously,
their context window size imposed a limit on the number of examples that can be
shown, making example selection techniques crucial for identifying the
maximally effective set of examples. However, the recent advent of Long Context
Language Models (LCLMs) has significantly increased the number of examples that
can be included in context, raising an important question of whether ICL
performance in a many-shot regime is still sensitive to the method of sample
selection. To answer this, we revisit these approaches in the context of LCLMs
through extensive experiments on 18 datasets spanning 4 tasks. Surprisingly, we
observe that sophisticated example selection techniques do not yield
significant improvements over a simple random sample selection method. Instead,
we find that the advent of LCLMs has fundamentally shifted the challenge of ICL
from that of selecting the most effective examples to that of collecting
sufficient examples to fill the context window. Specifically, in certain
datasets, including all available examples does not fully utilize the context
window; however, by augmenting the examples in context with a simple data
augmentation approach, we substantially improve ICL performance by 5%.
Long Context Language Models reduce the sensitivity of In-Context Learning to example selection techniques and instead emphasize the importance of sufficient and augmented examples.
In-Context Learning Long Context Language Models