As the model size grows, according to the scal-
ing law, more training data is required, raising con-
cerns about running out of high-quality training
data (Villalobos et al., 2022; Hoffmann et al., 2022).
One straightforward way to overcome this issue is
to train on data repeatedly. However, data repeti-
tion notoriously leads to performance degradation
as discussed above. Motivated by this contradic-
tion, several works study the consequence of pre-
training on datasets repeatedly for multiple epochs.
Muennighoff et al. (2023) find that with constrained
data and fixed compute budgets, repeatedly training
on the same data up to 4 epochs yields negligible
changes to loss compared to training on unique data.
They also propose a scaling law accounting for the
diminishing of returns with more repetition and
excess parameters. Xue et al. (2023) also observe
a multi-epoch degradation in model performance
and find that dataset size, model parameters, and
DataManagementForLargeLanguageModels-ASurvey