From Zero to Hero: How local curvature at artless initial conditions leads away from bad minima
<|reference_start|>From Zero to Hero: How local curvature at artless initial conditions leads away from bad minima: We provide an analytical study of the evolution of the Hessian during gradient descent dynamics, and relate a transition in its spectral properties to the ability of finding good minima. We focus on the phase retrieval problem as a case study for complex loss landscapes. We first characterize the high-dimensional limit where both the number $M$ and the dimension $N$ of the data are going to infinity at fixed signal-to-noise ratio $\alpha = M/N$. For small $\alpha$, the Hessian is uninformative with respect to the signal. For $\alpha$ larger than a critical value, the Hessian displays at short-times a downward direction pointing towards good minima. While descending, a transition in the spectrum takes place: the direction is lost and the system gets trapped in bad minima. Hence, the local landscape is benign and informative at first, before gradient descent brings the system into a uninformative maze. Through both theoretical analysis and numerical experiments, we show that this dynamical transition plays a crucial role for finite (even very large) $N$: it allows the system to recover the signal well before the algorithmic threshold corresponding to the $N\rightarrow\infty$ limit. Our analysis sheds light on this new mechanism that facilitates gradient descent dynamics in finite dimensions, and highlights the importance of a good initialization based on spectral properties for optimization in complex high-dimensional landscapes.<|reference_end|>
@article{bonnaire2024from,
    title={From Zero to Hero: How local curvature at artless initial conditions
  leads away from bad minima},
    author={Tony Bonnaire, Giulio Biroli, Chiara Cammarota},
    journal={arXiv preprint arXiv:2403.02418},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.02418},
    primaryClass={cs.LG cond-mat.dis-nn cond-mat.stat-mech}
}