title: RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder
    Language Models
summary: In this paper, we investigate the in-context learning ability of
  retrieval-augmented encoder-decoder language models. We first conduct a
  comprehensive analysis of the state-of-the-art ATLAS model and identify its
  limitations in in-context learning, primarily due to a mismatch between
  pretraining and testing, as well as a restricted context length. To address
  these issues, we propose RAVEN, a model that combines retrieval-augmented
  masked language modeling and prefix language modeling. We further introduce
  Fusion-in-Context Learning to enhance the few-shot performance by enabling the
  model to leverage more in-context examples without requiring additional
  training or model modifications. Through extensive experiments, we demonstrate
  that RAVEN significantly outperforms ATLAS and achieves results comparable to
  the most advanced language models in certain scenarios, despite having
  substantially fewer parameters. Our work underscores the potential of
  retrieval-augmented encoder-decoder language models for in-context learning and
  encourages further research in this direction.
paper_authors._id: 64dc8dd9476b8dd78b124d30
paper_authors.name: Jie Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-17T13:53:19.507Z
paper_authors._id: 64dd0d55304349e047edd96c
paper_authors.avatarUrl: /avatars/b5bc04180353fb9ef1474f01185050d3.svg
paper_authors.fullname: Jie Huang
paper_authors.type: user
paper_authors.user: jeffhj
paper_authors._id: 64dc8dd9476b8dd78b124d31
paper_authors.name: Wei Ping
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-15T08:00:35.505Z
paper_authors._id: 661cdb08ed360bdc4f6db37c
paper_authors.avatarUrl: /avatars/bced64e9b7f46125a03d6f6e66ae7e33.svg
paper_authors.fullname: Wei Ping
paper_authors.type: user
paper_authors.user: wpingnet
paper_authors._id: 64dc8dd9476b8dd78b124d32
paper_authors.name: Peng Xu
paper_authors._id: 64dc8dd9476b8dd78b124d33
paper_authors.name: Mohammad Shoeybi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:20:24.428Z
paper_authors._id: 6641544c695975af2cbd0da6
paper_authors.avatarUrl: /avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
paper_authors.fullname: Mohammad Shoeybi
paper_authors.type: user
paper_authors.user: shoeybi
paper_authors._id: 64dc8dd9476b8dd78b124d34
paper_authors.name: Kevin Chen-Chuan Chang
paper_authors._id: 64dc8dd9476b8dd78b124d35
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-16T14:16:03.617Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_title: RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder
    Language Models
paper_summary: In this paper, we investigate the in-context learning ability of
  retrieval-augmented encoder-decoder language models. We first conduct a
  comprehensive analysis of the state-of-the-art ATLAS model and identify its
  limitations in in-context learning, primarily due to a mismatch between
  pretraining and testing, as well as a restricted context length. To address
  these issues, we propose RAVEN, a model that combines retrieval-augmented
  masked language modeling and prefix language modeling. We further introduce
  Fusion-in-Context Learning to enhance the few-shot performance by enabling the
  model to leverage more in-context examples without requiring additional
  training or model modifications. Through extensive experiments, we demonstrate
  that RAVEN significantly outperforms ATLAS and achieves results comparable to
  the most advanced language models in certain scenarios, despite having
  substantially fewer parameters. Our work underscores the potential of
  retrieval-augmented encoder-decoder language models for in-context learning and
  encourages further research in this direction.
paper_ai_summary: RAVEN enhances in-context learning in retrieval-augmented encoder-decoder models by combining masked and prefix language modeling, achieving superior performance with fewer parameters.
paper_ai_keywords: retrieval-augmented encoder-decoder ATLAS model in-context learning masked language modeling prefix language modeling Fusion-in-Context Learning few-shot performance