• “Clean” samples, with less than 20% token contamination,
• “Not clean” samples, with greater than (or equal to) 20% token contamination,
• “Not dirty” samples, with less than 80% token contamination,
• “Dirty” samples, with greater than (or equal to) 80% token contamination.

There is an additional confounding factor that we attempt to address directly. With the given definition of
contamination (as well as other definitions mentioned in the literature), there is a possibility that a sample

75

7B

70B

Dataset

HellaSwag (L = 40)

Model Subset Type Avg. Contam. %

Zn
-5.73
9.56
-2.27
7.42
-5.46
9.17
-2.06
6.84
-4.08
9.71
-3.50
9.80
-2.75
6.50
-2.25
6.49
-2.00
4.64
-2.75
7.87
Table 51: Contamination analysis results for affected datasets. No other evaluation datasets had sufficient
evidence to be considered affected by contamination. Avg. Contam. % denotes the average per-sample
contamination percentage for the given subset type. Models sizes refer to pretrained-only models
Llama2