On the other hand, following the discovery of the scal-
ing law, there has been a rapid increase in model parameters,
making autoregressive models the mainstream. Researchers
are also exploring whether larger models can be pretrained
using the RAG approach. RETRO++[Wang et al., 2023a], an

Figure 4: Taxonomy of RAG’s Core Components

extension of RETRO, increased the model’s parameter scale.
Studies have found consistent improvements in text genera-
tion quality, factual accuracy, low toxicity, and downstream
task accuracy, particularly in knowledge-intensive tasks such
as open-domain question answering. These research findings
highlight the promising direction of pretraining autoregres-
sive language models in conjunction with retrieval for future
foundational models.
Retrieval-AugmentedGenerationforLargeLanguageModels-ASurvey