More Room for Language: Investigating the Effect of Retrieval on Language Models
<|reference_start|>More Room for Language: Investigating the Effect of Retrieval on Language Models: Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually relevant information that could aid the language modeling objective. We introduce an 'ideal retrieval' methodology to study these models in a fully controllable setting. We conduct an extensive evaluation to examine how retrieval augmentation affects the behavior of the underlying language model. Among other things, we observe that these models: i) save substantially less world knowledge in their weights, ii) are better at understanding local context and inter-word dependencies, but iii) are worse at comprehending global context.<|reference_end|>
@article{samuel2024more,
    title={More Room for Language: Investigating the Effect of Retrieval on
  Language Models},
    author={David Samuel, Lucas Georges Gabriel Charpentier and Sondre Wold},
    journal={arXiv preprint arXiv:2404.10939},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10939},
    primaryClass={cs.CL}
}