  Modern neural networks are undeniably successful. Numerous works study how
the curvature of loss landscapes can affect the quality of solutions. In this
work we study the loss landscape by considering the Hessian matrix during
network training with large learning rates - an attractive regime that is
(in)famously unstable. We characterise the instabilities of gradient descent,
and we observe the striking phenomena of \textit{landscape flattening} and
\textit{landscape shift}, both of which are intimately connected to the
instabilities of training.

Machine Learning