k(cid:88)

j=1

retrieval

6.1 RAG in Augmentation Stages
As a knowledge-intensive task, RAG employs different tech-
nical approaches during the language model training’s pre-
training, fine-tuning, and inference stages.
Pre-training Stage
Since the emergence of pre-trained models, researchers have
delved into enhancing the performance of Pre-trained Lan-
guage Models (PTMs) in open-domain Question Answering
(QA) through retrieval methods at the pre-training stage. Rec-
ognizing and expanding implicit knowledge in pre-trained
models can be challenging. REALM[Arora et al., 2023] in-
troduces a more modular and interpretable knowledge em-
bedding approach. Following the Masked Language Model
(MLM) paradigm, REALM models both pre-training and
fine-tuning as a retrieve-then-predict process, where the lan-
guage model pre-trains by predicting masked tokens y based
on masked sentences x, modeling P (x|y).
RETRO[Borgeaud et al., 2022]leverages
Retrieval-AugmentedGenerationforLargeLanguageModels-ASurvey