Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features
<|reference_start|>Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features: We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing the difference between test risk curves of pure gradient and stochastic gradient flows. We apply the general theory to a simple model of weak features, which displays the double descent phenomenon, and explicitly compute the corrections brought about by the added stochastic term in the dynamics, as a function of time and model parameters. The analytical results are compared to simulations of discrete-time stochastic gradient descent and show good agreement.<|reference_end|>
@article{veiga2024stochastic,
    title={Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution
  for Weak Features},
    author={Rodrigo Veiga, Anastasia Remizova, Nicolas Macris},
    journal={arXiv preprint arXiv:2402.07626},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.07626},
    primaryClass={stat.ML cond-mat.dis-nn cs.LG}
}