Effect of Retrieving more documents Models are trained with either 5 or 10 retrieved latent
documents, and we do not observe signiﬁcant differences in performance between them. We have the
ﬂexibility to adjust the number of retrieved documents at test time, which can affect performance and
runtime. Figure 3 (left) shows that retrieving more documents at test time monotonically improves
Open-domain QA results for RAG-Sequence, but performance peaks for RAG-Token at 10 retrieved
documents. Figure 3 (right) shows that retrieving more documents leads to higher Rouge-L for
RAG-Token at the expense of Bleu-1, but the effect is less pronounced for RAG-Sequence.

Figure 3: Left: NQ performance as more documents are retrieved. Center: Retrieval recall perfor-
mance in NQ. Right: MS-MARCO Bleu-1 and Rouge-L as more documents are retrieved.

5 Related Work
Retrieval-AugmentedGenerationfor Knowledge-IntensiveNLPTasks