title: Ambiguity-Aware In-Context Learning with Large Language Models
summary: In-context learning (ICL) i.e. showing LLMs only a few task-specific
  demonstrations has led to downstream gains with no task-specific fine-tuning
  required. However, LLMs are sensitive to the choice of prompts, and therefore a
  crucial research question is how to select good demonstrations for ICL. One
  effective strategy is leveraging semantic similarity between the ICL
  demonstrations and test inputs by using a text retriever, which however is
  sub-optimal as that does not consider the LLM's existing knowledge about that
  task. From prior work (Min et al., 2022), we already know that labels paired
  with the demonstrations bias the model predictions. This leads us to our
  hypothesis whether considering LLM's existing knowledge about the task,
  especially with respect to the output label space can help in a better
  demonstration selection strategy. Through extensive experimentation on three
  text classification tasks, we find that it is beneficial to not only choose
  semantically similar ICL demonstrations but also to choose those demonstrations
  that help resolve the inherent label ambiguity surrounding the test example.
  Interestingly, we find that including demonstrations that the LLM previously
  mis-classified and also fall on the test example's decision boundary, brings
  the most performance gain.
paper_authors._id: 6503c370b6247620e03a959b
paper_authors.name: Lingyu Gao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-15T02:43:02.937Z
paper_authors._id: 63e161721e5a4f34b7a27cf8
paper_authors.avatarUrl: /avatars/e711755020b591baafb3071684bf470c.svg
paper_authors.fullname: Lingyu Gao
paper_authors.type: user
paper_authors.user: lygao
paper_authors._id: 6503c370b6247620e03a959c
paper_authors.name: Aditi Chaudhary
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-11-15T23:08:13.624Z
paper_authors._id: 65554f55f0cf8603f3672344
paper_authors.avatarUrl: /avatars/fe466a428db842506d7a569a64c46567.svg
paper_authors.fullname: Aditi
paper_authors.type: user
paper_authors.user: NLP-Data
paper_authors._id: 6503c370b6247620e03a959d
paper_authors.name: Krishna Srinivasan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-15T09:20:40.080Z
paper_authors._id: 6266ea037cc13cb44d35a168
paper_authors.avatarUrl: /avatars/096d479f403cb561bab1dc13f0fc0339.svg
paper_authors.fullname: Krishna Srinivasan
paper_authors.type: user
paper_authors.user: krishna2
paper_authors._id: 6503c370b6247620e03a959e
paper_authors.name: Kazuma Hashimoto
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-26T02:46:19.308Z
paper_authors._id: 662b133cf688584a05424b53
paper_authors.avatarUrl: /avatars/50fa3307613ab47c0907e0cddce457ad.svg
paper_authors.fullname: Kazuma Hashimoto
paper_authors.type: user
paper_authors.user: hassyGo
paper_authors._id: 6503c370b6247620e03a959f
paper_authors.name: Karthik Raman
paper_authors._id: 6503c370b6247620e03a95a0
paper_authors.name: Michael Bendersky
paper_title: Ambiguity-Aware In-Context Learning with Large Language Models
paper_summary: In-context learning (ICL) i.e. showing LLMs only a few task-specific
  demonstrations has led to downstream gains with no task-specific fine-tuning
  required. However, LLMs are sensitive to the choice of prompts, and therefore a
  crucial research question is how to select good demonstrations for ICL. One
  effective strategy is leveraging semantic similarity between the ICL
  demonstrations and test inputs by using a text retriever, which however is
  sub-optimal as that does not consider the LLM's existing knowledge about that
  task. From prior work (Min et al., 2022), we already know that labels paired
  with the demonstrations bias the model predictions. This leads us to our
  hypothesis whether considering LLM's existing knowledge about the task,
  especially with respect to the output label space can help in a better
  demonstration selection strategy. Through extensive experimentation on three
  text classification tasks, we find that it is beneficial to not only choose
  semantically similar ICL demonstrations but also to choose those demonstrations
  that help resolve the inherent label ambiguity surrounding the test example.
  Interestingly, we find that including demonstrations that the LLM previously
  mis-classified and also fall on the test example's decision boundary, brings
  the most performance gain.
paper_ai_summary: In-context learning benefits from selecting demonstrations that resolve label ambiguity and include examples on the decision boundary, leveraging the LLM's existing knowledge.
paper_ai_keywords: in-context learning ICL LLMs task-specific demonstrations semantic similarity text retriever label space label ambiguity decision boundary mis-classified