How Does Generative Retrieval Scale to Millions of Passages?
Popularized by the Differentiable Search Index, the emerging paradigm of
generative retrieval re-frames the classic information retrieval problem into a
sequence-to-sequence modeling task, forgoing external indices and encoding an
entire document corpus within a single Transformer. Although many different
approaches have been proposed to improve the effectiveness of generative
retrieval, they have only been evaluated on document corpora on the order of
100k in size. We conduct the first empirical study of generative retrieval
techniques across various corpus scales, ultimately scaling up to the entire MS
MARCO passage ranking task with a corpus of 8.8M passages and evaluating model
sizes up to 11B parameters. We uncover several findings about scaling
generative retrieval to millions of passages; notably, the central importance
of using synthetic queries as document representations during indexing, the
ineffectiveness of existing proposed architecture modifications when accounting
for compute cost, and the limits of naively scaling model parameters with
respect to retrieval performance. While we find that generative retrieval is
competitive with state-of-the-art dual encoders on small corpora, scaling to
millions of passages remains an important and unsolved challenge. We believe
these findings will be valuable for the community to clarify the current state
of generative retrieval, highlight the unique challenges, and inspire new
research directions.
646afce31556443f24ba0d73
Ronak Pradeep
claimed_verified
2024-11-08T12:47:16.384Z
5ec824d4968f6028e0559f6f
https://cdn-avatars.huggingface.co/v1/production/uploads/5ec824d4968f6028e0559f6f/b7C7D41DTmVOa0DHJmSE9.jpeg
Ronak Pradeep
user
ronak
646afce31556443f24ba0d74
Kai Hui
646afce31556443f24ba0d75
Jai Gupta
646afce31556443f24ba0d76
Adam D. Lelkes
646afce31556443f24ba0d77
Honglei Zhuang
646afce31556443f24ba0d78
Jimmy Lin
646afce31556443f24ba0d79
Donald Metzler
646afce31556443f24ba0d7a
Vinh Q. Tran
claimed_verified
2023-05-22T18:01:28.955Z
646b7ede628e5b50b2de8d5a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/y_be_9O4FO1sbRiNKzuJd.jpeg
Vinh Q. Tran
user
vqtran
How Does Generative Retrieval Scale to Millions of Passages?
Popularized by the Differentiable Search Index, the emerging paradigm of
generative retrieval re-frames the classic information retrieval problem into a
sequence-to-sequence modeling task, forgoing external indices and encoding an
entire document corpus within a single Transformer. Although many different
approaches have been proposed to improve the effectiveness of generative
retrieval, they have only been evaluated on document corpora on the order of
100k in size. We conduct the first empirical study of generative retrieval
techniques across various corpus scales, ultimately scaling up to the entire MS
MARCO passage ranking task with a corpus of 8.8M passages and evaluating model
sizes up to 11B parameters. We uncover several findings about scaling
generative retrieval to millions of passages; notably, the central importance
of using synthetic queries as document representations during indexing, the
ineffectiveness of existing proposed architecture modifications when accounting
for compute cost, and the limits of naively scaling model parameters with
respect to retrieval performance. While we find that generative retrieval is
competitive with state-of-the-art dual encoders on small corpora, scaling to
millions of passages remains an important and unsolved challenge. We believe
these findings will be valuable for the community to clarify the current state
of generative retrieval, highlight the unique challenges, and inspire new
research directions.
Generative retrieval scales to large corpora with significant findings on the importance of synthetic queries and the limits of parameter scaling.
generative retrieval sequence-to-sequence Transformer synthetic queries MS MARCO passage ranking dual encoders