four components: knowledge encoding, memory, retrieval
and generation. Given an input query x, we first retrieve
K possibly helpful entries M = {m1,··· , mK} from the
memory corpora M. Each m is a memory entry containing
the encoded single key embedding and a sequence of value
embeddings (we will describe how to encode knowledge
items into memory entries in Sec. 3.2). With it, the retriever
can use embedding similarity to find relevant memory entries.
We model this retrieval process as sampling from distribu-
tion p(M | x). Then, we condition on both the retrieved set
M and the original input query x to generate the output y,
modeled as p(y | x, M ). To obtain the overall likelihood of
generating y, we treat M as a latent variable from the entire
memory ˜M and marginalize over it yielding:

p(y | x) =

(cid:88)

M⊂ ˜M

p(M | x)

· p(y | x, M )

.

(cid:124)

(cid:123)(cid:122)

(cid:125)

(cid:124)

(cid:123)(cid:122)

(cid:125)

retrieval

generation

(1)
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory