2.2.1 Deduplication
Deduplication is widely used in data management
procedures of many prominent LLMs and prepro-
cessing of publicly available datasets (Brown et al.,
2020; Workshop et al., 2022; Touvron et al., 2023a;
Raffel et al., 2020). Lee et al. (2021) use N-gram
similarity with MinHash (Broder, 1997) to detect
duplications in training datasets and find that dedu-
plication is beneficial in memorization mitigation,
train-test overlap avoidance, and training efficiency

while keeping model perplexity. Kandpal et al.
(2022) also show that deduplication can consid-
erably lower the success rate of privacy attacks
aiming at model memorization.
DataManagementForLargeLanguageModels-ASurvey