RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder
  Language Models
In this paper, we investigate the in-context learning ability of
retrieval-augmented encoder-decoder language models. We first conduct a
comprehensive analysis of the state-of-the-art ATLAS model and identify its
limitations in in-context learning, primarily due to a mismatch between
pretraining and testing, as well as a restricted context length. To address
these issues, we propose RAVEN, a model that combines retrieval-augmented
masked language modeling and prefix language modeling. We further introduce
Fusion-in-Context Learning to enhance the few-shot performance by enabling the
model to leverage more in-context examples without requiring additional
training or model modifications. Through extensive experiments, we demonstrate
that RAVEN significantly outperforms ATLAS and achieves results comparable to
the most advanced language models in certain scenarios, despite having
substantially fewer parameters. Our work underscores the potential of
retrieval-augmented encoder-decoder language models for in-context learning and
encourages further research in this direction.
64dc8dd9476b8dd78b124d30
Jie Huang
claimed_verified
2023-08-17T13:53:19.507Z
64dd0d55304349e047edd96c
/avatars/b5bc04180353fb9ef1474f01185050d3.svg
Jie Huang
user
jeffhj
64dc8dd9476b8dd78b124d31
Wei Ping
claimed_verified
2024-04-15T08:00:35.505Z
661cdb08ed360bdc4f6db37c
/avatars/bced64e9b7f46125a03d6f6e66ae7e33.svg
Wei Ping
user
wpingnet
64dc8dd9476b8dd78b124d32
Peng Xu
64dc8dd9476b8dd78b124d33
Mohammad Shoeybi
claimed_verified
2024-05-28T07:20:24.428Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
64dc8dd9476b8dd78b124d34
Kevin Chen-Chuan Chang
64dc8dd9476b8dd78b124d35
Bryan Catanzaro
admin_assigned
2023-08-16T14:16:03.617Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder
  Language Models
In this paper, we investigate the in-context learning ability of
retrieval-augmented encoder-decoder language models. We first conduct a
comprehensive analysis of the state-of-the-art ATLAS model and identify its
limitations in in-context learning, primarily due to a mismatch between
pretraining and testing, as well as a restricted context length. To address
these issues, we propose RAVEN, a model that combines retrieval-augmented
masked language modeling and prefix language modeling. We further introduce
Fusion-in-Context Learning to enhance the few-shot performance by enabling the
model to leverage more in-context examples without requiring additional
training or model modifications. Through extensive experiments, we demonstrate
that RAVEN significantly outperforms ATLAS and achieves results comparable to
the most advanced language models in certain scenarios, despite having
substantially fewer parameters. Our work underscores the potential of
retrieval-augmented encoder-decoder language models for in-context learning and
encourages further research in this direction.
RAVEN enhances in-context learning in retrieval-augmented encoder-decoder models by combining masked and prefix language modeling, achieving superior performance with fewer parameters.
retrieval-augmented encoder-decoder ATLAS model in-context learning masked language modeling prefix language modeling Fusion-in-Context Learning few-shot performance