ACM Comput. Surv., Vol. 1, No. 1, Article . Publication date: February 2022.

8

Ziwei Ji, et al.

Another problematic scenario is when duplicates from the dataset are not properly filtered out. It
is almost impossible to check hundreds of gigabytes of text corpora manually. Lee et al. [96] show
that duplicated examples from the pretraining corpus bias the model to favor generating repeats of
the memorized phrases from the duplicated examples.
SurveyofHallucinationinNatural Language Generation