We pre-train our model on the Web-Image-Text
dataset [51], a large-scale corpus containing 3 billion image
alt-text caption pairs collected from the public Web. Since
the dataset is noisy, we add a filter to remove data points
whose captions are shorter than 50 characters. This yields
roughly 1.3 billion image caption pairs for pre-training.
We denote the pre-training Web-Image-Text dataset [51]
as D. We use the text generation objective used in Wang et
al. [45]) to pre-train our model on D. Given an image-text
example x =(img, txt) from D, we randomly sample a
prefix length Tp. We feed x<Tp that contains the text prefix
and image to the model as input and our objective is to
generate x≥Tp containing the rest of the text as output. The
training goal is to condition on x<Tp and autoregressively
generate the remaining text sequence x≥Tp:

LPrefixLM = − Ex∼D(cid:2)log p(x≥Tp | x<Tp )(cid:3)
(cid:105)

(cid:104) (cid:88)

(4)

log p(xi | x<i)

.

= − Ex∼D

i≥Tp
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory