Scaling Law in the Data-Constrained Regime. What if the training data are limited? The work [188] examines a data-
constrained regime and observes that, for a fixed compute budget, training with up to 4 epochs of repeated data yields
negligible changes in the objective function compared to a single epoch3. However, further increasing the number of epochs
will lead to a decrease in the performance of the learned model.
3Unlike general machine learning models, which are typically trained with multiple epochs, it is common practice to train Language Models (LLMs) using a
couple of epochs [24, 138].

7

Efficient LLM Algorithmic Survey, Nov, 2023, USA.

Ding, Chen, et al.
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey