pη(z|x)

pθ(yi|x, z, y1:i−1)

z∈top-k(p(·|x))

z∈top-k(p(·|x))

(cid:88)

N(cid:89)

i

RAG-Token Model
In the RAG-Token model we can draw a different latent document for each
target token and marginalize accordingly. This allows the generator to choose content from several
documents when producing an answer. Concretely, the top K documents are retrieved using the
retriever, and then the generator produces a distribution for the next output token for each document,
before marginalizing, and repeating the process with the following output token, Formally, we deﬁne:

pRAG-Token(y|x) ≈ N(cid:89)

(cid:88)

i

z∈top-k(p(·|x))

pη(z|x)pθ(yi|x, z, y1:i−1)

Finally, we note that RAG can be used for sequence classiﬁcation tasks by considering the target class
as a target sequence of length one, in which case RAG-Sequence and RAG-Token are equivalent.

2.2 Retriever: DPR
The retrieval component pη(z|x) is based on DPR [26]. DPR follows a bi-encoder architecture:
Retrieval-AugmentedGenerationfor Knowledge-IntensiveNLPTasks