Pre-Training Pipeline. We first train the multimodal
retriever on our modified version of the Wikipedia Image
Text (WIT) dataset via Lcontra. We use the Adafactor op-
timizer without momentum (β1 = 0, β2 = 0.999), with
weight decay of 0.0012, and with a peak learning rate of
6e4, to train for 10 epochs. We use this checkpoint to warm-
start our generative pre-training. We set the number of re-
trieved knowledge entries as K = 10 during pre-training,
and use adafactor with a peak learning rate of 1e−3 and in-
verse squared root learning rate scheduler with 10,000 linear
warm-up steps. We use LP ref ixLM as the main objective,
adding Lcontra, Ldecor and Lalign weighted by 0.01. We
use a batch size of 4096 across 256 CloudTPUv4 chips and
train for about 5 days.

5. Experimental Results
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory