2.5 Decoding
At test time, RAG-Sequence and RAG-Token require different ways to approximate arg maxy p(y|x).
RAG-Token The RAG-Token model can be seen as a standard, autoregressive seq2seq genera-
z∈top-k(p(·|x)) pη(zi|x)pθ(yi|x, zi, y1:i−1) To
tor with transition probability: p(cid:48)
decode, we can plug p(cid:48)
RAG-Sequence For RAG-Sequence, the likelihood p(y|x) does not break into a conventional per-
token likelihood, hence we cannot solve it with a single beam search. Instead, we run beam search for
each document z, scoring each hypothesis using pθ(yi|x, z, y1:i−1). This yields a set of hypotheses
Y , some of which may not have appeared in the beams of all documents. To estimate the probability
of an hypothesis y we run an additional forward pass for each document z for which y does not
appear in the beam, multiply generator probability with pη(z|x) and then sum the probabilities across
Retrieval-AugmentedGenerationfor Knowledge-IntensiveNLPTasks