Visualizing the Loss Landscape of Neural Nets
Neural network training relies on our ability to find ````````"good" minimizers of highly non-convex loss functions. It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple ``"filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture effects the loss landscape, and how training parameters affect the shape of minimizers.
 This paper provides visualizations of different deep network loss surfaces using 2D contour plots, both at minima and along optimization trajectories. They mention some subtle details that must be taken into account, such as scaling the plot axes by the filter magnitudes, in order to obtain correctly scaled plots. Overall, I think there is potential with this work but it feels preliminary. The visualizations are interesting and provide some general intuition, but they don*t yield any clear novel insights that could be used in practice. Also, several parts of the paper spend too much time on describing other work or on implementation details which could be moved to the appendix. General Comments: - I think Sections 2, 3, 4 are too long, we only start getting to the results section at the end of page 4. I suggest shortening Section 2, and it should be possible to combine Sections 3 and 4 into a page at most. 1D interpolations and 2D contour plots can be described in a few sentences each. - I think Section 5 can be put in the Appendix - it*s essentially an illustration of why the weight scaling is important. Once these details are done correctly, the experiments support the relatively well-accepted hypothesis that flat minima generalize better. - The plots in Section 6 are interesting, it would be nice if the authors had an explanation of why the loss surface changes the way it does when skip connections are added. - In Section 7, it*s less useful to spend time describing what happens when the visualization is done wrong (i.e. projecting along random directions rather than PCA vectors) - this can be put in the Appendix. I would suggest just including the visualizations of the optimization trajectories which are done correctly and focus on deriving interesting/useful conclusions from them.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Visualizing the Loss Landscape of Neural Nets
Neural network training relies on our ability to find ````````"good" minimizers of highly non-convex loss functions. It is well known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effect on the underlying loss landscape, is not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple ``"filter normalization" method that helps us visualize loss function curvature, and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture effects the loss landscape, and how training parameters affect the shape of minimizers.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper provides visualizations of different deep network loss surfaces using 2D contour plots, both at minima and along optimization trajectories. They mention some subtle details that must be taken into account, such as scaling the plot axes by the filter magnitudes, in order to obtain correctly scaled plots. Overall, I think there is potential with this work but it feels preliminary. The visualizations are interesting and provide some general intuition, but they don*t yield any clear novel insights that could be used in practice. Also, several parts of the paper spend too much time on describing other work or on implementation details which could be moved to the appendix. General Comments: - I think Sections 2, 3, 4 are too long, we only start getting to the results section at the end of page 4. I suggest shortening Section 2, and it should be possible to combine Sections 3 and 4 into a page at most. 1D interpolations and 2D contour plots can be described in a few sentences each. - I think Section 5 can be put in the Appendix - it*s essentially an illustration of why the weight scaling is important. Once these details are done correctly, the experiments support the relatively well-accepted hypothesis that flat minima generalize better. - The plots in Section 6 are interesting, it would be nice if the authors had an explanation of why the loss surface changes the way it does when skip connections are added. - In Section 7, it*s less useful to spend time describing what happens when the visualization is done wrong (i.e. projecting along random directions rather than PCA vectors) - this can be put in the Appendix. I would suggest just including the visualizations of the optimization trajectories which are done correctly and focus on deriving interesting/useful conclusions from them.
assistant
 This paper provides visualizations of different deep network loss surfaces using 2D contour plots, both at minima and along optimization trajectories. They mention some subtle details that must be taken into account, such as scaling the plot axes by the filter magnitudes, in order to obtain correctly scaled plots. Overall, I think there is potential with this work but it feels preliminary. The visualizations are interesting and provide some general intuition, but they don*t yield any clear novel insights that could be used in practice. Also, several parts of the paper spend too much time on describing other work or on implementation details which could be moved to the appendix. General Comments: - I think Sections 2, 3, 4 are too long, we only start getting to the results section at the end of page 4. I suggest shortening Section 2, and it should be possible to combine Sections 3 and 4 into a page at most. 1D interpolations and 2D contour plots can be described in a few sentences each. - I think Section 5 can be put in the Appendix - it*s essentially an illustration of why the weight scaling is important. Once these details are done correctly, the experiments support the relatively well-accepted hypothesis that flat minima generalize better. - The plots in Section 6 are interesting, it would be nice if the authors had an explanation of why the loss surface changes the way it does when skip connections are added. - In Section 7, it*s less useful to spend time describing what happens when the visualization is done wrong (i.e. projecting along random directions rather than PCA vectors) - this can be put in the Appendix. I would suggest just including the visualizations of the optimization trajectories which are done correctly and focus on deriving interesting/useful conclusions from them.