embarrassingly parallel nature of the task, we are able to find all such 10-grams (and their full lengths) in
our entire dataset in around seven hours (including time to tokenize), utilizing an estimated 1,500 cores.
As there are many confounding factors at play when determining whether dataset contamination has
contributed to evaluation performance (mostly stemming from the fact that "clean" and "dirty" subsets do
not necessarily well-estimate the population distribution), we make the following assumption: In the event
of dataset contamination contributing to evaluation performance, we expect both the "cleanest" examples to
have an overall worse average score than their complement, and the "dirtiest" samples to have an overall better
average score than their complement. It is insufficient evidence for contamination if only one of these were
true. To this end, we define four (non-disjoint) subset types as follows:
Llama2