Meta-training with Demonstration Retrieval for Efficient Few-shot
  Learning
Large language models show impressive results on few-shot NLP tasks. However,
these models are memory and computation-intensive. Meta-training allows one to
leverage smaller models for few-shot generalization in a domain-general and
task-agnostic manner; however, these methods alone results in models that may
not have sufficient parameterization or knowledge to adapt quickly to a large
variety of tasks. To overcome this issue, we propose meta-training with
demonstration retrieval, where we use a dense passage retriever to retrieve
semantically similar labeled demonstrations to each example for more varied
supervision. By separating external knowledge from model parameters, we can use
meta-training to train parameter-efficient models that generalize well on a
larger variety of tasks. We construct a meta-training set from UnifiedQA and
CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our
knowledge, our work is the first to combine retrieval with meta-training, to
use DPR models to retrieve demonstrations, and to leverage demonstrations from
many tasks simultaneously, rather than randomly sampling demonstrations from
the training set of the target task. Our approach outperforms a variety of
targeted parameter-efficient and retrieval-augmented few-shot methods on QA,
NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our
approach can be meta-trained and fine-tuned quickly on a single GPU.
64a3808c225b2121f6ae80f6
Aaron Mueller
extracted_pending
2023-07-04T02:14:36.640Z
60a5abbc8b5faf05f3c4d757
/avatars/eba6c3e61a7a6a12f487d531cf7a9934.svg
Aaron Mueller
user
amueller
64a3808c225b2121f6ae80f7
Kanika Narang
64a3808c225b2121f6ae80f8
Lambert Mathias
64a3808c225b2121f6ae80f9
Qifan Wang
64a3808c225b2121f6ae80fa
Hamed Firooz
Meta-training with Demonstration Retrieval for Efficient Few-shot
  Learning
Large language models show impressive results on few-shot NLP tasks. However,
these models are memory and computation-intensive. Meta-training allows one to
leverage smaller models for few-shot generalization in a domain-general and
task-agnostic manner; however, these methods alone results in models that may
not have sufficient parameterization or knowledge to adapt quickly to a large
variety of tasks. To overcome this issue, we propose meta-training with
demonstration retrieval, where we use a dense passage retriever to retrieve
semantically similar labeled demonstrations to each example for more varied
supervision. By separating external knowledge from model parameters, we can use
meta-training to train parameter-efficient models that generalize well on a
larger variety of tasks. We construct a meta-training set from UnifiedQA and
CrossFit, and propose a demonstration bank based on UnifiedQA tasks. To our
knowledge, our work is the first to combine retrieval with meta-training, to
use DPR models to retrieve demonstrations, and to leverage demonstrations from
many tasks simultaneously, rather than randomly sampling demonstrations from
the training set of the target task. Our approach outperforms a variety of
targeted parameter-efficient and retrieval-augmented few-shot methods on QA,
NLI, and text classification tasks (including SQuAD, QNLI, and TREC). Our
approach can be meta-trained and fine-tuned quickly on a single GPU.
Meta-training with demonstration retrieval enhances few-shot NLP performance by leveraging dense passage retrievers and external knowledge sources.
meta-training demonstration retrieval dense passage retriever parameter-efficient models UnifiedQA CrossFit demonstration bank few-shot methods QA NLI text classification SQuAD QNLI TREC