Table 4. Visual Question Answering results on A-OKVQA.

5.1. Evaluating on Knowledge-Based VQA

One of the most knowledge intensive visual-language
tasks is knowledge-based visual question answering (VQA),
exemplified by the OK-VQA [29] and A-OKVQA [35]
benchmarks. To finetune our pre-trained model on these
VQA tasks, we use the same generative objective where the
model takes in an image question pair as input and gener-
ates the text answer as output. There are a few differences
between the fine-tuning and the pre-training stages: 1) we
set the number of retrieved knowledge entries to K = 50, so
the model is able to retrieve sufficient supporting evidence;
2) we freeze the whole base V-L encoder to stabilize train-
ing; and 3) we use a batch size of 128, with the Adafactor
optimizier, a peak learning rate of 1e-4. We use the soft
VQA accuracy metric [3] to evaluate the model’s generated
answer.
REVEAL-Retrieval-AugmentedVisual-LanguagePre-Trainingwith Multi-SourceMultimodalKnowledgeMemory