Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks
<|reference_start|>Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks: This paper presents a comprehensive analysis of critical point sets in two-layer neural networks. To study such complex entities, we introduce the critical embedding operator and critical reduction operator as our tools. Given a critical point, we use these operators to uncover the whole underlying critical set representing the same output function, which exhibits a hierarchical structure. Furthermore, we prove existence of saddle branches for any critical set whose output function can be represented by a narrower network. Our results provide a solid foundation to the further study of optimization and training behavior of neural networks.<|reference_end|>
@article{zhang2024geometry,
    title={Geometry of Critical Sets and Existence of Saddle Branches for Two-layer
  Neural Networks},
    author={Leyang Zhang, Yaoyu Zhang, Tao Luo},
    journal={arXiv preprint arXiv:2405.17501},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17501},
    primaryClass={cs.LG math.OC}
}