title: In-Context Principle Learning from Mistakes
summary: In-context learning (ICL, also known as few-shot prompting) has been the
  standard method of adapting LLMs to downstream tasks, by learning from a few
  input-output examples. Nonetheless, all ICL-based approaches only learn from
  correct input-output pairs. In this paper, we revisit this paradigm, by
  learning more from the few given input-output examples. We introduce Learning
  Principles (LEAP): First, we intentionally induce the model to make mistakes on
  these few examples; then we reflect on these mistakes, and learn explicit
  task-specific "principles" from them, which help solve similar problems and
  avoid common mistakes; finally, we prompt the model to answer unseen test
  questions using the original few-shot examples and these learned general
  principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop
  question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning,
  and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the
  strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and
  Claude-2.1. For example, LEAP improves over the standard few-shot prompting
  using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does
  not require any more input or examples than the standard few-shot prompting
  settings.
paper_authors._id: 65c5915184ed215272356bcd
paper_authors.name: Tianjun Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-01T05:49:17.162Z
paper_authors._id: 6374cd6b6ea8da14f8fef8dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
paper_authors.fullname: Tianjun Zhang
paper_authors.type: user
paper_authors.user: tianjunz
paper_authors._id: 65c5915184ed215272356bce
paper_authors.name: Aman Madaan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:32:23.977Z
paper_authors._id: 61bbc2b2d96f5d4111901dcb
paper_authors.avatarUrl: /avatars/83c826d5cad7bab27d33585cb7e64575.svg
paper_authors.fullname: Aman Madaan
paper_authors.type: user
paper_authors.user: amanmadaan
paper_authors._id: 65c5915184ed215272356bcf
paper_authors.name: Luyu Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:32:30.523Z
paper_authors._id: 6009aa9126e23ab5edb47afd
paper_authors.avatarUrl: /avatars/f8afa9cf648227ad5817323451ede378.svg
paper_authors.fullname: Luyu Gao
paper_authors.type: user
paper_authors.user: Luyu
paper_authors._id: 65c5915184ed215272356bd0
paper_authors.name: Steven Zheng
paper_authors._id: 65c5915184ed215272356bd1
paper_authors.name: Swaroop Mishra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:32:56.676Z
paper_authors._id: 61841feb7c380c66d40cad54
paper_authors.avatarUrl: /avatars/7a79362a24b844860781fb0625527eae.svg
paper_authors.fullname: Swaroop Mishra
paper_authors.type: user
paper_authors.user: swaroop7
paper_authors._id: 65c5915184ed215272356bd2
paper_authors.name: Yiming Yang
paper_authors._id: 65c5915184ed215272356bd3
paper_authors.name: Niket Tandon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:33:20.842Z
paper_authors._id: 62e69a709bb6d949274ad32b
paper_authors.avatarUrl: /avatars/b6a796d12a1c1c708f5b46c9082cdee6.svg
paper_authors.fullname: Niket Tandon
paper_authors.type: user
paper_authors.user: nikett
paper_authors._id: 65c5915184ed215272356bd4
paper_authors.name: Uri Alon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:33:39.720Z
paper_authors._id: 62cd8153248f9e6bc20ab250
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657635637504-62cd8153248f9e6bc20ab250.jpeg
paper_authors.fullname: Uri Alon
paper_authors.type: user
paper_authors.user: urialon
paper_title: In-Context Principle Learning from Mistakes
paper_summary: In-context learning (ICL, also known as few-shot prompting) has been the
  standard method of adapting LLMs to downstream tasks, by learning from a few
  input-output examples. Nonetheless, all ICL-based approaches only learn from
  correct input-output pairs. In this paper, we revisit this paradigm, by
  learning more from the few given input-output examples. We introduce Learning
  Principles (LEAP): First, we intentionally induce the model to make mistakes on
  these few examples; then we reflect on these mistakes, and learn explicit
  task-specific "principles" from them, which help solve similar problems and
  avoid common mistakes; finally, we prompt the model to answer unseen test
  questions using the original few-shot examples and these learned general
  principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop
  question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning,
  and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the
  strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and
  Claude-2.1. For example, LEAP improves over the standard few-shot prompting
  using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does
  not require any more input or examples than the standard few-shot prompting
  settings.
paper_ai_keywords: in-context learning (ICL) few-shot prompting LLMs (Large Language Models) input-output examples Learning Principles (LEAP) task-specific principles multi-hop question answering (Hotpot QA) textual QA (DROP) Big-Bench Hard reasoning math problems (GSM8K and MATH) GPT-3.5-turbo GPT-4 GPT-4 turbo Claude-2.1