Ambiguity-Aware In-Context Learning with Large Language Models
In-context learning (ICL) i.e. showing LLMs only a few task-specific
demonstrations has led to downstream gains with no task-specific fine-tuning
required. However, LLMs are sensitive to the choice of prompts, and therefore a
crucial research question is how to select good demonstrations for ICL. One
effective strategy is leveraging semantic similarity between the ICL
demonstrations and test inputs by using a text retriever, which however is
sub-optimal as that does not consider the LLM's existing knowledge about that
task. From prior work (Min et al., 2022), we already know that labels paired
with the demonstrations bias the model predictions. This leads us to our
hypothesis whether considering LLM's existing knowledge about the task,
especially with respect to the output label space can help in a better
demonstration selection strategy. Through extensive experimentation on three
text classification tasks, we find that it is beneficial to not only choose
semantically similar ICL demonstrations but also to choose those demonstrations
that help resolve the inherent label ambiguity surrounding the test example.
Interestingly, we find that including demonstrations that the LLM previously
mis-classified and also fall on the test example's decision boundary, brings
the most performance gain.
6503c370b6247620e03a959b
Lingyu Gao
extracted_confirmed
2023-09-15T02:43:02.937Z
63e161721e5a4f34b7a27cf8
/avatars/e711755020b591baafb3071684bf470c.svg
Lingyu Gao
user
lygao
6503c370b6247620e03a959c
Aditi Chaudhary
extracted_pending
2023-11-15T23:08:13.624Z
65554f55f0cf8603f3672344
/avatars/fe466a428db842506d7a569a64c46567.svg
Aditi
user
NLP-Data
6503c370b6247620e03a959d
Krishna Srinivasan
admin_assigned
2023-09-15T09:20:40.080Z
6266ea037cc13cb44d35a168
/avatars/096d479f403cb561bab1dc13f0fc0339.svg
Krishna Srinivasan
user
krishna2
6503c370b6247620e03a959e
Kazuma Hashimoto
extracted_confirmed
2024-04-26T02:46:19.308Z
662b133cf688584a05424b53
/avatars/50fa3307613ab47c0907e0cddce457ad.svg
Kazuma Hashimoto
user
hassyGo
6503c370b6247620e03a959f
Karthik Raman
6503c370b6247620e03a95a0
Michael Bendersky
Ambiguity-Aware In-Context Learning with Large Language Models
In-context learning (ICL) i.e. showing LLMs only a few task-specific
demonstrations has led to downstream gains with no task-specific fine-tuning
required. However, LLMs are sensitive to the choice of prompts, and therefore a
crucial research question is how to select good demonstrations for ICL. One
effective strategy is leveraging semantic similarity between the ICL
demonstrations and test inputs by using a text retriever, which however is
sub-optimal as that does not consider the LLM's existing knowledge about that
task. From prior work (Min et al., 2022), we already know that labels paired
with the demonstrations bias the model predictions. This leads us to our
hypothesis whether considering LLM's existing knowledge about the task,
especially with respect to the output label space can help in a better
demonstration selection strategy. Through extensive experimentation on three
text classification tasks, we find that it is beneficial to not only choose
semantically similar ICL demonstrations but also to choose those demonstrations
that help resolve the inherent label ambiguity surrounding the test example.
Interestingly, we find that including demonstrations that the LLM previously
mis-classified and also fall on the test example's decision boundary, brings
the most performance gain.
In-context learning benefits from selecting demonstrations that resolve label ambiguity and include examples on the decision boundary, leveraging the LLM's existing knowledge.
in-context learning ICL LLMs task-specific demonstrations semantic similarity text retriever label space label ambiguity decision boundary mis-classified