title: Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
summary: Scaling laws with respect to the amount of training data and the number of
  parameters allow us to predict the cost-benefit trade-offs of pretraining
  language models (LMs) in different configurations. In this paper, we consider
  another dimension of scaling: the amount of data available at inference time.
  Specifically, we find that increasing the size of the datastore used by a
  retrieval-based LM monotonically improves language modeling and several
  downstream tasks without obvious saturation, such that a smaller model
  augmented with a large datastore outperforms a larger LM-only model on
  knowledge-intensive tasks. By plotting compute-optimal scaling curves with
  varied datastore, model, and pretraining data sizes, we show that using larger
  datastores can significantly improve model performance for the same training
  compute budget. We carry out our study by constructing a 1.4 trillion-token
  datastore named MassiveDS, which is the largest and the most diverse
  open-sourced datastore for retrieval-based LMs to date, and designing an
  efficient pipeline for studying datastore scaling in a computationally
  accessible manner. Finally, we analyze the effect of improving the retriever,
  datastore quality filtering, and other design choices on our observed scaling
  trends. Overall, our results show that datastore size should be considered as
  an integral part of LM efficiency and performance trade-offs. To facilitate
  future research, we open-source our datastore and code at
  https://github.com/RulinShao/retrieval-scaling.
paper_authors._id: 6699bf4e9fad95dc8e351184
paper_authors.name: Rulin Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:33.164Z
paper_authors._id: 6334a0bd31a2be3938c59537
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
paper_authors.fullname: Rulin Shao
paper_authors.type: user
paper_authors.user: rulins
paper_authors._id: 6699bf4e9fad95dc8e351185
paper_authors.name: Jacqueline He
paper_authors._id: 6699bf4e9fad95dc8e351186
paper_authors.name: Akari Asai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:23:20.147Z
paper_authors._id: 6266e0cb7a1f5a1562c4e86e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6266e0cb7a1f5a1562c4e86e/kkJZPssa76mdRPnKH-q3e.jpeg
paper_authors.fullname: Akari Asai
paper_authors.type: user
paper_authors.user: akariasai
paper_authors._id: 6699bf4e9fad95dc8e351187
paper_authors.name: Weijia Shi
paper_authors._id: 6699bf4e9fad95dc8e351188
paper_authors.name: Tim Dettmers
paper_authors._id: 6699bf4e9fad95dc8e351189
paper_authors.name: Sewon Min
paper_authors._id: 6699bf4e9fad95dc8e35118a
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 6699bf4e9fad95dc8e35118b
paper_authors.name: Pang Wei Koh
paper_title: Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
paper_summary: Scaling laws with respect to the amount of training data and the number of
  parameters allow us to predict the cost-benefit trade-offs of pretraining
  language models (LMs) in different configurations. In this paper, we consider
  another dimension of scaling: the amount of data available at inference time.
  Specifically, we find that increasing the size of the datastore used by a
  retrieval-based LM monotonically improves language modeling and several
  downstream tasks without obvious saturation, such that a smaller model
  augmented with a large datastore outperforms a larger LM-only model on
  knowledge-intensive tasks. By plotting compute-optimal scaling curves with
  varied datastore, model, and pretraining data sizes, we show that using larger
  datastores can significantly improve model performance for the same training
  compute budget. We carry out our study by constructing a 1.4 trillion-token
  datastore named MassiveDS, which is the largest and the most diverse
  open-sourced datastore for retrieval-based LMs to date, and designing an
  efficient pipeline for studying datastore scaling in a computationally
  accessible manner. Finally, we analyze the effect of improving the retriever,
  datastore quality filtering, and other design choices on our observed scaling
  trends. Overall, our results show that datastore size should be considered as
  an integral part of LM efficiency and performance trade-offs. To facilitate
  future research, we open-source our datastore and code at
  https://github.com/RulinShao/retrieval-scaling.
paper_ai_summary: Increasing the size of the datastore for retrieval-based language models improves performance without obvious saturation, suggesting datastore size is a key factor in model efficiency and performance.
paper_ai_keywords: retrieval-based LMs datastore language modeling downstream tasks compute-optimal scaling curves pretraining MassiveDS retriever datastore quality filtering