REPLUG [Shi et al., 2023] utilizes a retriever and an LLM
to calculate the probability distributions of the retrieved doc-
uments and then performs supervised training by computing
the KL divergence. This straightforward and effective train-
ing method enhances the performance of the retrieval model
by using an LM as the supervisory signal, eliminating the
need for specific cross-attention mechanisms.

UPRISE [Cheng et al., 2023a] also employs frozen LLMs
to fine-tune the prompt retriever. Both the LLM and the re-
triever take prompt-input pairs as inputs and utilize the scores
provided by the LLM to supervise the retriever’s training, ef-
fectively treating the LLM as a dataset labeler. In addition,
Atlas [Izacard et al., 2022] proposes four methods of super-
vised fine-tuning embedding models:

• Attention Distillation. This approach employs cross-
attention scores generated by the LLM during output to
distill the model’s knowledge.
RAG forLargeLanguageModels-ASurvey