Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity
<|reference_start|>Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity: One of the most intriguing findings in the structure of neural network landscape is the phenomenon of mode connectivity: For two typical global minima, there exists a path connecting them without barrier. This concept of mode connectivity has played a crucial role in understanding important phenomena in deep learning.   In this paper, we conduct a fine-grained analysis of this connectivity phenomenon. First, we demonstrate that in the overparameterized case, the connecting path can be as simple as a two-piece linear path, and the path length can be nearly equal to the Euclidean distance. This finding suggests that the landscape should be nearly convex in a certain sense. Second, we uncover a surprising star-shaped connectivity: For a finite number of typical minima, there exists a center on minima manifold that connects all of them simultaneously via linear paths. These results are provably valid for linear networks and two-layer ReLU networks under a teacher-student setup, and are empirically supported by models trained on MNIST and CIFAR-10.<|reference_end|>
@article{lin2024exploring,
    title={Exploring Neural Network Landscapes: Star-Shaped and Geodesic
  Connectivity},
    author={Zhanran Lin, Puheng Li, Lei Wu},
    journal={arXiv preprint arXiv:2404.06391},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06391},
    primaryClass={cs.LG stat.ML}
}