General Optimization Process
As part of the general optimization process, the training data
typically consists of input-output pairs, aiming to train the
model to produce the output y given the input x. In the work
of Self-Mem [Cheng et al., 2023b], a traditional
training
process is employed, where given the input x, relevant
documents z are retrieved (selecting Top-1 in the paper), and
after integrating (x, z), the model generates the output y.
The paper utilizes two common paradigms for fine-tuning,
namely Joint-Encoder and Dual-Encoder [Arora et al., 2023,
Wang et al., 2022b,
Lewis et al., 2020, Xia et al., 2019,
Cai et al., 2021, Cheng et al., 2022].
RAG forLargeLanguageModels-ASurvey