12 HALLUCINATION IN VISION-LANGUAGE GENERATION
With the vast advancement of the Transformer architecture [35, 189] in both CV and NLP, there
is a trend to pre-train large-scale unified vision-language (VL) models [1, 27, 105, 196, 197, 200]
to perform vision grounded text generation tasks, such as image captioning and visual question
answering (VQA). Generally, there are two common schemas for vision-language pre-training: 1)
pre-train from scratch with a massive amount of image-text pairs as well as optionally a large
text-only corpus; or 2) initialize model parameters from a large pre-trained LM and then adapt
it to the VL domain with adequate image-text pairs. Either way, the learned vision and language
representations are aligned in the same multimodal space and the resulting model can be seen as a
LM that understands visual information. Therefore, the hallucination problem is also observed in
VL models due to similar reasons as found in NLG.
SurveyofHallucinationinNatural Language Generation