Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks
<|reference_start|>Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks: In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size $\gamma$ and momentum parameter $\beta$ that allows us to identify an intrinsic quantity $\lambda = \frac{ \gamma }{ (1 - \beta)^2 }$ which uniquely defines the optimisation path and provides a simple acceleration rule. When training a $2$-layer diagonal linear network in an overparametrised regression setting, we characterise the recovered solution through an implicit regularisation problem. We then prove that small values of $\lambda$ help to recover sparse solutions. Finally, we give similar but weaker results for stochastic momentum gradient descent. We provide numerical experiments which support our claims.<|reference_end|>
@article{papazov2024leveraging,
    title={Leveraging Continuous Time to Understand Momentum When Training Diagonal
  Linear Networks},
    author={Hristo Papazov, Scott Pesme, Nicolas Flammarion},
    journal={arXiv preprint arXiv:2403.05293},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.05293},
    primaryClass={cs.LG math.OC stat.ML}
}