4.3. Implementation Details

Incorporating all

the components introduced above,
REVEAL can be directly pre-trained over large-scale im-
age caption datasets after proper initialization. As our model
architecture is based on T5 and ViT, we use pre-trained
ViT checkpoints from [50] and pre-trained T5 checkpoints
from [33] to initialize the encoder parameters. The query
head, key head and attentive fusion layers are initialized
from upper T5, while the base text encoder is initialized
from lower T5. The combination of these modules can be
found in Table 2 for three three model variants, REVEAL-
Base, REVEAL-Large and REVEAL, of which the largest
REVEAL model has around 2 billion parameters.

Knowledge Sources

Accuracy (%) Memory (GB)
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory