In the VL domain, the research on hallucination is still in its very early stage and how to measure
and mitigate hallucination is an open question. In this section, we first review the hallucination in
image captioning as it is the only VL task that has corresponding previous research works. Then,
we introduce hallucination phenomena found in other VL tasks. Finally, we discuss potential future
research directions on this problem.

ACM Comput. Surv., Vol. 1, No. 1, Article . Publication date: February 2022.

34

Ziwei Ji, et al.

12.1 Object Hallucination in Image Captioning

Definition. Object hallucination is defined as models generating captions that contain non-
existent or inaccurate objects from the input image. Following tasks in NLG, we also categorize
object hallucination into intrinsic and extrinsic ones:

Fig. 2. Examples of intrinsic and extrinsic object hallucination in image captioning.
SurveyofHallucinationinNatural Language Generation