remaining ones including the token embedding layer as ‘lower-layer’.

(cid:0)b(z)(cid:1) ∈ Rd is a d-dimensional embed-

EmbKey(z) = ϕKey
ding vector encoded via Key Head. Each value is a sequence
of token embeddings representing the full information of
knowledge item z. We follow a similar procedure as in [14]
to precompute key/value embeddings of knowledge items
from different sources and index them in a unified knowl-
edge memory. We continuously re-compute the memory
key/value embeddings as the model parameters get updated
during the pre-training phase. We update the memory ˜M
asynchronously at every 1000 training steps.
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory