adding a discrete retrieval step to neural networks (Miller
et al., 2016; Chen et al., 2017), but did not apply the frame-
work to language model pre-training and employed non-
learned retrievers to handle large-scale document collec-
tions. In the language modeling literature, the k-Nearest
Neighbor Language Model (Khandelwal et al., 2019) (kNN-
LM) retrieves similar LM examples to improve memoriza-
tion. However, kNN-LM was not ﬁne-tuned for downstream
tasks, perhaps because it is unclear how to adapt the re-
trieval mechanism: a kNN can only use examples labeled
for the target task—during ﬁne-tuning, this precludes LM
examples, which contain the desired world knowledge. In
contrast, REALM’s retriever is designed to transfer to other
tasks, and the retrieval is just text, not a labeled example.
We evaluate our approach by ﬁne-tuning the mod-
els pre-trained with REALM on the task of Open-
domain Question Answering (Open-QA), one of the most
REALM