Therefore, we propose to use the Perceiver architec-
ture [19] as the Value Head to encode and compress knowl-
edge items. The Perceiver model uses a transformer de-
coder ψ(·) with learnable c-length latent embeddings to
compress the full token sequence into an arbitrary length
c, such that EmbValue(z) = ψ(b(z)) ∈ Rc×d (In our ex-
periments we use c = 32). This lets us retrieve top-K
memory entries for K as large as a hundred. To make the
compressed embeddings generated by Perceiver more ex-
pressive, we add two additional regularizations. The first
one is a disentangled regularization [16] that forces every
two output tokens to be linearly de-correlated Ldecor =
, and the sec-
ond one is an alignment regularization that minimizes the
distance of L2-Norm between the query and compressed
knowledge embedding: Lalign =

(cid:13)(cid:13)(cid:13)Covariance(cid:0)ψ(cid:0)b(zi)(cid:1), ψ(cid:0)b(zj)(cid:1)(cid:1)(cid:13)(cid:13)(cid:13)2
z∥ψ(cid:0)b(z)(cid:1)∥2
(cid:80)
(cid:80)
x∥b(x)∥2
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory