Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling
<|reference_start|>Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling: The remarkable success of large language pretraining and the discovery of scaling laws signify a paradigm shift in machine learning. Notably, the primary objective has evolved from minimizing generalization error to reducing approximation error, and the most effective strategy has transitioned from regularization (in a broad sense) to scaling up models. This raises a critical question:   Do the established principles that proved successful in the generalization-centric era remain valid in this new era of scaling?   This paper examines several influential regularization-based principles that may no longer hold true in the scaling-centric, large language model (LLM) era. These principles include explicit L2 regularization and implicit regularization through small batch sizes and large learning rates. Additionally, we identify a new phenomenon termed ``scaling law crossover,'' where two scaling curves intersect at a certain scale, implying that methods effective at smaller scales may not generalize to larger ones. Together, these observations highlight two fundamental questions within this new paradigm:   $\bullet$ Guiding Principles for Scaling: If regularization is no longer the primary guiding principle for model design, what new principles are emerging to guide scaling?   $\bullet$ Model Comparison at Scale: How to reliably and effectively compare models at the scale where only a single experiment is feasible?<|reference_end|>
@article{xiao2024rethinking,
    title={Rethinking Conventional Wisdom in Machine Learning: From Generalization
  to Scaling},
    author={Lechao Xiao},
    journal={arXiv preprint arXiv:2409.15156},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.15156},
    primaryClass={cs.LG stat.ML}
}