H = Encoder(x[SEP ]m)

hi = Decoder(CrossAttn(H), y < i)

(6)

PGξ (.|x, y < i) = Sof tmax(hi)

(7)
For the Dual-Encoder, the system establishes two indepen-
dent encoders, each responsible for encoding the input (query,
context) and the document, respectively. The output is then
subject to bidirectional cross-attention processing by the de-
coder in sequence. The authors choose to use the Transformer
[Vaswani et al., 2017] as the building block for both architec-
tures and optimize Gξ Negative Log-Likelihood (NLL) loss.
Hx = SourceEncoder(x)Hm = M emoryEncoder(x)

(8)

(9)

(10)

hi = Decoder(CrossAttn(Hx, Hm), y < i)

Lnll = −

logPGξ (yt|x, m, y < t)

|y|(cid:88)

t=1
Retrieval-AugmentedGenerationforLargeLanguageModels-ASurvey