title: Meta-training with Demonstration Retrieval for Efficient Few-shot
    Learning
summary: Large language models show impressive results on few-shot NLP tasks. However,
  these models are memory and computation-intensive. Meta-training allows one to
  leverage smaller models for few-shot generalization in a domain-general and
  task-agnostic manner; however, these methods alone results in models that may
  not have sufficient parameterization or knowledge to adapt quickly to a large
  variety of tasks. To overcome this issue, we propose meta-training with
  demonstration retrieval, where we use a dense passage retriever to retrieve
  semantically similar labeled demonstrations to each example for more varied
  supervision. By separating external knowledge from model parameters, we can use
  meta-training to train parameter-efficient models that generalize well on a
  larger variety of tasks. We construct a meta-training set from UnifiedQA and
  CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our
  knowledge, our work is the first to combine retrieval with meta-training, to
  use DPR models to retrieve demonstrations, and to leverage demonstrations from
  many tasks simultaneously, rather than randomly sampling demonstrations from
  the training set of the target task. Our approach outperforms a variety of
  targeted parameter-efficient and retrieval-augmented few-shot methods on QA,
  NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our
  approach can be meta-trained and fine-tuned quickly on a single GPU.
paper_authors._id: 64a3808c225b2121f6ae80f6
paper_authors.name: Aaron Mueller
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-07-04T02:14:36.640Z
paper_authors._id: 60a5abbc8b5faf05f3c4d757
paper_authors.avatarUrl: /avatars/eba6c3e61a7a6a12f487d531cf7a9934.svg
paper_authors.fullname: Aaron Mueller
paper_authors.type: user
paper_authors.user: amueller
paper_authors._id: 64a3808c225b2121f6ae80f7
paper_authors.name: Kanika Narang
paper_authors._id: 64a3808c225b2121f6ae80f8
paper_authors.name: Lambert Mathias
paper_authors._id: 64a3808c225b2121f6ae80f9
paper_authors.name: Qifan Wang
paper_authors._id: 64a3808c225b2121f6ae80fa
paper_authors.name: Hamed Firooz
paper_title: Meta-training with Demonstration Retrieval for Efficient Few-shot
    Learning
paper_summary: Large language models show impressive results on few-shot NLP tasks. However,
  these models are memory and computation-intensive. Meta-training allows one to
  leverage smaller models for few-shot generalization in a domain-general and
  task-agnostic manner; however, these methods alone results in models that may
  not have sufficient parameterization or knowledge to adapt quickly to a large
  variety of tasks. To overcome this issue, we propose meta-training with
  demonstration retrieval, where we use a dense passage retriever to retrieve
  semantically similar labeled demonstrations to each example for more varied
  supervision. By separating external knowledge from model parameters, we can use
  meta-training to train parameter-efficient models that generalize well on a
  larger variety of tasks. We construct a meta-training set from UnifiedQA and
  CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our
  knowledge, our work is the first to combine retrieval with meta-training, to
  use DPR models to retrieve demonstrations, and to leverage demonstrations from
  many tasks simultaneously, rather than randomly sampling demonstrations from
  the training set of the target task. Our approach outperforms a variety of
  targeted parameter-efficient and retrieval-augmented few-shot methods on QA,
  NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our
  approach can be meta-trained and fine-tuned quickly on a single GPU.
paper_ai_summary: Meta-training with demonstration retrieval enhances few-shot NLP performance by leveraging dense passage retrievers and external knowledge sources.
paper_ai_keywords: meta-training demonstration retrieval dense passage retriever parameter-efficient models UnifiedQA CrossFit demonstration bank few-shot methods QA NLI text classification SQuAD QNLI TREC