Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding
<|reference_start|>Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding: In this paper, we investigate the loss landscape of one-hidden-layer neural networks with ReLU-like activation functions trained with the empirical squared loss. As the activation function is non-differentiable, it is so far unclear how to completely characterize the stationary points. We propose the conditions for stationarity that apply to both non-differentiable and differentiable cases. Additionally, we show that, if a stationary point does not contain "escape neurons", which are defined with first-order conditions, then it must be a local minimum. Moreover, for the scalar-output case, the presence of an escape neuron guarantees that the stationary point is not a local minimum. Our results refine the description of the saddle-to-saddle training process starting from infinitesimally small (vanishing) initialization for shallow ReLU-like networks, linking saddle escaping directly with the parameter changes of escape neurons. Moreover, we are also able to fully discuss how network embedding, which is to instantiate a narrower network within a wider network, reshapes the stationary points.<|reference_end|>
@article{wu2024loss,
    title={Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points,
  Saddle Escaping, and Network Embedding},
    author={Zhengqing Wu, Berfin Simsek, Francois Ged},
    journal={arXiv preprint arXiv:2402.05626},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.05626},
    primaryClass={cs.LG}
}