Statistical law in machine learning
In machine learning, a neural scaling law is an empirical scaling law that describes how neural network performance changes as key factors are scaled up or down. These factors typically include the number of parameters, training dataset size, and training cost.
Introduction.
In general, a deep learning model can be characterized by four parameters: model size, training dataset size, training cost, and the post-training error rate (e.g., the test set error rate). Each of these variables can be defined as a real number, usually written as formula_1 (respectively: parameter count, dataset size, computing cost, and loss).
A neural scaling law is a theoretical or empirical statistical law between these parameters. There are also other parameters with other scaling laws.
Size of the model.
In most cases, the model's size is simply the number of parameters. However, one complication arises with the use of sparse models, such as mixture-of-expert models. With sparse models, during inference, only a fraction of their parameters are used. In comparison, most other kinds of neural networks, such as transformer models, always use all their parameters during inference.
Size of the training dataset.
The size of the training dataset is usually quantified by the number of data points within it. Larger training datasets are typically preferred, as they provide a richer and more diverse source of information from which the model can learn. This can lead to improved generalization performance when the model is applied to new, unseen data. However, increasing the size of the training dataset also increases the computational resources and time required for model training.
With the "pretrain, then finetune" method used for most large language models, there are two kinds of training dataset: the "pretraining" dataset and the "finetuning" dataset. Their sizes have different effects on model performance. Generally, the finetuning dataset is less than 1% the size of pretraining dataset.
In some cases, a small amount of high quality data suffices for finetuning, and more data does not necessarily improve performance.
Cost of training.
Training cost is typically measured in terms of time (how long it takes to train the model) and computational resources (how much processing power and memory are required). It is important to note that the cost of training can be significantly reduced with efficient training algorithms, optimized software libraries, and parallel computing on specialized hardware such as GPUs or TPUs.
The cost of training a neural network model is a function of several factors, including model size, training dataset size, the training algorithm complexity, and the computational resources available. In particular, doubling the training dataset size does not necessarily double the cost of training, because one may train the model for several times over the same dataset (each being an "epoch").
Performance.
The performance of a neural network model is evaluated based on its ability to accurately predict the output given some input data. Common metrics for evaluating model performance include:
Performance can be improved by using more data, larger models, different training algorithms, regularizing the model to prevent overfitting, and early stopping using a validation set.
Examples.
(Hestness, Narang, et al, 2017).
The 2017 paper is a common reference point for neural scaling laws fitted by statistical analysis on experimental data. Previous works before the 2000s, as cited in the paper, were either theoretical or orders of magnitude smaller in scale. Whereas previous works generally found the scaling exponent to scale like formula_2, with formula_3, the paper found that formula_4.
Of the factors they varied, only task can change the exponent formula_5. Changing the architecture optimizers, regularizers, and loss functions, would only change the proportionality factor, not the exponent. For example, for the same task, one architecture might have formula_6 while another might have formula_7. They also found that for a given architecture, the number of parameters necessary to reach lowest levels of loss, given a fixed dataset size, grows like formula_8 for another exponent formula_9.
They studied machine translation with LSTM (formula_10), generative language modelling with LSTM (formula_11), ImageNet classification with ResNet (formula_12), and speech recognition with two hybrid (LSTMs complemented by either CNNs or an attention decoder) architectures (formula_13).
(Henighan, Kaplan, et al, 2020).
A 2020 analysis studied statistical relations between formula_14 over a wide range of values and found similar scaling laws, over the range of formula_15, formula_16, and over multiple modalities (text, video, image, text to image, etc.).
In particular, the scaling laws it found are (Table 1 of ):
The scaling law of formula_42 was confirmed during the training of GPT-3 (Figure 3.1 ).
Chinchilla scaling (Hoffmann, et al, 2022).
One particular scaling law ("Chinchilla scaling") states that, for a large language model (LLM) autoregressively trained for one epoch, with a cosine learning rate schedule, we have:formula_43where the variables are
and the statistical parameters are
Although Besiroglu et. al. claims that the statistical estimation is slightly off, and should be formula_55.
The statistical laws were fitted over experimental data with formula_56.
Since there are 4 variables related by 2 equations, imposing 1 additional constraint and 1 additional optimization objective allows us to solve for all four variables. In particular, for any fixed formula_29, we can uniquely solve for all 4 variables that minimizes formula_47. This provides us with the optimal formula_59 for any fixed formula_29:formula_61Plugging in the numerical values, we obtain the "Chinchilla efficient" model size and training dataset size, as well as the test loss achievable:formula_62Similarly, we may find the optimal training dataset size and training compute budget for any fixed model parameter size, and so on.
There are other estimates for "Chinchilla efficient" model size and training dataset size. The above is based on a statistical model of formula_63. One can also directly fit a statistical law for formula_59 without going through the detour, for which one obtains:formula_65or as tabulated:
Discrepancy.
The Chinchilla scaling law analysis for training transformer language models suggests that for a given training compute budget (formula_29), to achieve the minimal pretraining loss for that budget, the number of model parameters (formula_24) and the number of training tokens (formula_18) should be scaled in equal proportions, formula_69. 
This conclusion differs from analysis conducted by Kaplan et al., which found that formula_24 should be increased more quickly than formula_18, formula_72.
This discrepancy can primarily be attributed to the studies using different methods for measuring model size; Kaplan et al. counted only non-embedding paramters, which when analyzed at smaller model sizes leads to biased coefficients.
Secondary effects also arise due to differences in hyperparameter tuning and learning rate schedules.
Beyond Chinchilla scaling.
As Chinchilla scaling has been the reference point for many large-scaling training runs, there had been a concurrent effort to go "beyond Chinchilla scaling", meaning to modify some of the training pipeline in order to obtain the same loss with less effort, or deliberately train for longer than what is "Chinchilla optimal".
Usually, the goal is to make the scaling law exponent larger, which means the same loss can be trained for much less compute. For instance, filtering data can make the scaling law exponent larger.
Another strand of research studies how to deal with limited data, as according to Chinchilla scaling laws, the training dataset size for the largest language models already approaches what is available on the internet. found that augmenting the dataset with a mix of "denoising objectives" constructed from the dataset improves performance. studies optimal scaling when all available data is already exhausted (such as in rare languages), so one must train multiple epoches over the same dataset (whereas Chinchilla scaling requires only one epoch). The Phi series of small language models were trained on textbook-like data generated by large language models, for which data is only limited by amount of compute available.
Chinchilla optimality was defined as "optimal for training compute", whereas in actual production-quality models, there will be a lot of inference after training is complete. "Overtraining" during training means better performance during inference. LLaMA models were overtrained for this reason. Subsequent studies discovered scaling laws in the overtraining regime, for dataset sizes up to 32x more than Chinchilla-optimal.
Broken neural scaling laws (BNSL).
A 2022 analysis found that many scaling behaviors of artificial neural networks follow a smoothly broken power law functional form:
formula_73
in which formula_21 refers to the quantity being scaled (i.e. formula_29, formula_24, formula_18, number of training steps, number of inference steps, or model input size) and formula_78 refers to the "downstream" (or upstream) performance evaluation metric of interest (e.g. prediction error, cross entropy, calibration error, AUROC, BLEU score percentage, F1 score, reward, Elo rating, solve rate, or FID score) in zero-shot, prompted, or fine-tuned settings. The parameters formula_79 are found by statistical fitting.
On a log–log plot, when formula_80 is not too large and formula_81 is subtracted out from the y-axis, this functional form looks like a series of linear segments connected by arcs; the formula_82 transitions between the segments are called "breaks", hence the name "broken neural scaling laws (BNSL)".
The scenarios in which the scaling behaviors of artificial neural networks were found to follow this functional form include large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, out-of-distribution detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, arithmetic, emergent abilities, double descent, supervised learning, unsupervised/self-supervised learning, and reinforcement learning (single agent and multi-agent).
The architectures for which the scaling behaviors of artificial neural networks were found to follow this functional form include residual neural networks, transformers, MLPs, , recurrent neural networks, convolutional neural networks, graph neural networks, U-nets, encoder-decoder (and encoder-only) (and decoder-only) models, ensembles (and non-ensembles), MoE (mixture of experts) (and non-MoE) models, and sparse pruned (and non-sparse unpruned) models.
Inference scaling.
Other than scaling up training compute, one can also scale up inference compute (or "test-time compute"). As an example, the Elo rating of AlphaGo improves steadily as it is allowed to spend more time on its Monte Carlo Tree Search per play.Fig 4 For AlphaGo Zero, increasing Elo by 120 requires either 2x model size and training, or 2x test-time search. Similarly, a language model for solving competition-level coding challenges, AlphaCode, consistently improved (log-linearly) in performance with more search time.
For Hex, 10x training-time compute trades for 15x test-time compute. For "Libratus" for heads up no-limit Texas hold 'em, and "Cicero" for "Diplomacy", and many other abstract games of partial information, inference-time searching improves performance at a similar tradeoff ratio, for up to 100,000x effective increase in training-time compute.
In 2024, the OpenAI o1 report documented that o1's performance consistently improved with both increased train-time compute and test-time compute, and gave numerous examples of test-time compute scaling in mathematics, scientific reasoning, and coding tasks.
One method for scaling up test-time compute is process-based supervision, where a model generates a step-by-step reasoning chain to answer a question, and another model (either human or AI) provides a reward score on some of the intermediate steps, not just the final answer. Process-based supervision can be scaled arbitrarily by using synthetic reward score without another model, for example, by running Monte Carlo rollouts and scoring each step in the reasoning according to how likely it leads to the right answer. Another method is by revision models, which are models trained to solve a problem multiple times, each time revising the previous attempt.
Other examples.
Vision transformers.
Vision transformers, similar to language transformers, exhibit scaling laws. A 2022 research trained vision transformers, with parameter counts formula_83, on image sets of sizes formula_84, for computing formula_85 (in units of TPUv3-core-days).
After training the model, it is finetuned on ImageNet training set. Let formula_47 be the error probability of the finetuned model classifying ImageNet test set. They found formula_87.
Neural machine translation.
Ghorbani, Behrooz et al. studied scaling laws for neural machine translation (specifically, English as source, and German as target) in encoder-decoder Transformer models, trained until convergence on the same datasets (thus they did not fit scaling laws for computing cost formula_29 or dataset size formula_18). They varied formula_90 They found three results:
The authors hypothesize that source-natural datasets have uniform and dull target sentences, and so a model that is trained to predict the target sentences would quickly overfit.
 trained Transformers for machine translations with sizes formula_101 on dataset sizes formula_102. They found the Kaplan et al (2020) scaling law applied to machine translation: formula_103. They also found the BLEU score scaling as formula_104.
Transfer learning.
Hernandez, Danny et al. studied scaling laws for transfer learning in language models. They trained a family of Transformers in three ways:
The idea is that pretraining on English should help the