Visualizing the Loss Landscape of Neural Nets
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and wellchosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.
Overview: Visualizing the loss landscape of the parameters in neural nets is hard, as the dimension of parameters in neural network is huge. Being able to visualize the loss landscape in neural net could help us understand how different techniques are helping to shape the loss surface, and how the loss surface "sharpness" vs "flatness" are related to generalization error.  Previous works include interpolating a 1D loss path between the parameters of two models to reveal the loss surface along the paths. However, 1D visualization sometimes can be misleading and losing critical information about the surrounding loss surface and local minimums. For 2D visualizations, usually 2 directions are selected for interpolating the loss surface along the 2 directions from an origin. However, as the neural net has a "scale-invariant" problem, which means the weights in different layers can appear in different orders but remain the same effect.  The author proposed a new way of visualizing neural network loss surface in 2D with consideration of scale-invariant in neural network parameters. The method is to sample two Gaussian random vectors that normalized by the filter norms in the neural network, and then compute the loss across different combinations of the two filter-normalized Gaussian random vectors from the minimizer. The author conducted experiments to show that the loss surface in 1D could be misleading, and their 2D filter-normalized plot is revealing more information about minima comparing to non-filter-normalized plot. The experiments on resnet also visualized the network trained under-different settings to show that certain techniques do help flatten the loss surface, and flattened surface is helping the generalization error.  Quality: The method used in the paper is technically sound and the experiments are well organized.   Clarity: The writing of the paper is mostly clear and easy-to-understand. There are two suggestions: 1) When introducing the method to generate filter-normalized plot, it is a little bit ambiguous when the author mention the term "filter". My understanding is that each part of the elements in the gaussian vector should be normalized by the corresponding convolutional filter in the neural network. Probably a simple toy example will be better explaining this idea. 2) The organization of the sections is a little bit ambiguous, as the author firstly introduced the method to generate 2D filter-normalized plot, but then in the first parts of section 3, the author showed the result for 1D plot to reveal the constraint of 1D plot. This is a little bit confusing. Moving the 1D plot to the front and introducing the method to generate 2D-filter-normalized plot could probably be better in the reading flow.  Originality: The major difference from the previous method is the 2D filter-normalized plot. However, the author also claimed that without filter-normalization, the plot could still reveal some of the local-minima of the two different methods. The other originality is to apply the visualization on the network trained by different methods.   Significance: The visualizations of the loss landscapes are very impressing and very informative. However, the method itself is a little bit simple. The observations and conclusions are very helpful, but stayed on an eyeball based analysis. If somehow the author could bring a more quantitative study of the sharpness/flatness of the landscape, or even relate the sharpness/flatness of the surface to the regularizer in training, or relate that to the sensitivity analysis of the network could be more impactful. Also, this method is still computational heavy, as one needs to compute the value of each interpolation block. I don't see high-resolution with an efficient implementation in their study._x000c_