SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
<|reference_start|>SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training: The effectiveness of large language models (LLMs) is often hindered by duplicated data in their extensive pre-training datasets. Current approaches primarily focus on detecting and removing duplicates, which risks the loss of valuable information and neglects the varying degrees of duplication. To address this, we propose a soft deduplication method that maintains dataset integrity while selectively reducing the sampling weight of data with high commonness. Central to our approach is the concept of "data commonness", a metric we introduce to quantify the degree of duplication by measuring the occurrence probabilities of samples using an n-gram model. Empirical analysis shows that this method significantly improves training efficiency, achieving comparable perplexity scores with at least a 26% reduction in required training steps. Additionally, it enhances average few-shot downstream accuracy by 1.77% when trained for an equivalent duration. Importantly, this approach consistently improves performance, even on rigorously deduplicated datasets, indicating its potential to complement existing methods and become a standard pre-training process for LLMs.<|reference_end|>
@article{he2024softdedup:,
    title={SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language
  Model Pre-training},
    author={Nan He, Weichen Xiong, Hanwen Liu, Yi Liao, Lei Ding, Kai Zhang,
  Guohua Tang, Xiao Han, Wei Yang},
    journal={arXiv preprint arXiv:2407.06654},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.06654},
    primaryClass={cs.CL cs.AI}
}