Deep Ensembles: A Loss Landscape Perspective
Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.
This paper is trying to answer the question why ensembles of deep neural networks trained with random initialization work so well in practice in improving accuracy. Their proposed hypothesis is that networks trained from different initializations, although all converge to a low-loss/high accuracy optimum, explore different modes in function space and therefore provide more diversity. To experimentally support their hypothesis, first they show that functions along a single training trajectory are similar, however trajectories starting from different initializations may significantly differ. The difference in function space is based on the fraction of points on which the two functions disagree in terms of their prediction. Second, they use different subspace sampling methods around a single optimum and demonstrate that they are significantly less diverse (low disagreement between predictions) than sampling from independent optima through diversity vs accuracy plots. Moreover, they comment on the recent observation that local optima are connected by low-loss tunnels. They experimentally show that even though low-loss/high accuracy path exists between local optima, these tunnels do not correspond to similar solutions in function space, further supporting the multi-mode hypothesis. The authors compare the relative benefit of subspace sampling, weight averaging and ensembling on accuracy and interpret their findings in terms of the hypothesis. Overall, the paper is very well written and provides interesting insights into the multi-modal structure of deep neural network loss landscapes. Even though the hypothesis of the paper is not entirely new and has been touched upon in Fort & Jastrzebski (2019), this paper contributes to the field by providing thorough experimental support and clear exposition of the idea. Therefore, I would accept this paper if the authors provided additional experimental results on a different dataset. The paper mentions that the trends are consistent across all datasets the authors have explored. However, they only provide results on CIFAR-10 (and a limited set of experiments on ImageNet). Since the contribution of the paper heavily relies on providing experimental verification, it would be important to include at least the diversity vs. accuracy plot for the other datasets they have explored to demonstrate that this phenomenon is not specific to CIFAR-10. Additionally, I would like to add a couple of comments on the paper that are not part of my decision, but could potentially improve the paper. -The diversity score introduced in the paper is simple and intuitive, however it would be interesting to see whether the results hold across different notions of function space disagreement. -It is mentioned in the paper that data augmentation has been used for training the ResNet20 architecture. Would the results change significantly without data augmentation, as it adds another source of randomness to the training procedure. -Some comments on the figures: in Figure 3 it is very difficult to discern any difference between different shades of red (disagreement values), and in this form the plots are not too informative. Maybe rescaling or a different way of presentation would help. Interpreting Figure 7/a is a bit difficult, probably a 3D plot would be useful to explain the different line sections.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Ensembles: A Loss Landscape Perspective
Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper is trying to answer the question why ensembles of deep neural networks trained with random initialization work so well in practice in improving accuracy. Their proposed hypothesis is that networks trained from different initializations, although all converge to a low-loss/high accuracy optimum, explore different modes in function space and therefore provide more diversity. To experimentally support their hypothesis, first they show that functions along a single training trajectory are similar, however trajectories starting from different initializations may significantly differ. The difference in function space is based on the fraction of points on which the two functions disagree in terms of their prediction. Second, they use different subspace sampling methods around a single optimum and demonstrate that they are significantly less diverse (low disagreement between predictions) than sampling from independent optima through diversity vs accuracy plots. Moreover, they comment on the recent observation that local optima are connected by low-loss tunnels. They experimentally show that even though low-loss/high accuracy path exists between local optima, these tunnels do not correspond to similar solutions in function space, further supporting the multi-mode hypothesis. The authors compare the relative benefit of subspace sampling, weight averaging and ensembling on accuracy and interpret their findings in terms of the hypothesis. Overall, the paper is very well written and provides interesting insights into the multi-modal structure of deep neural network loss landscapes. Even though the hypothesis of the paper is not entirely new and has been touched upon in Fort & Jastrzebski (2019), this paper contributes to the field by providing thorough experimental support and clear exposition of the idea. Therefore, I would accept this paper if the authors provided additional experimental results on a different dataset. The paper mentions that the trends are consistent across all datasets the authors have explored. However, they only provide results on CIFAR-10 (and a limited set of experiments on ImageNet). Since the contribution of the paper heavily relies on providing experimental verification, it would be important to include at least the diversity vs. accuracy plot for the other datasets they have explored to demonstrate that this phenomenon is not specific to CIFAR-10. Additionally, I would like to add a couple of comments on the paper that are not part of my decision, but could potentially improve the paper. -The diversity score introduced in the paper is simple and intuitive, however it would be interesting to see whether the results hold across different notions of function space disagreement. -It is mentioned in the paper that data augmentation has been used for training the ResNet20 architecture. Would the results change significantly without data augmentation, as it adds another source of randomness to the training procedure. -Some comments on the figures: in Figure 3 it is very difficult to discern any difference between different shades of red (disagreement values), and in this form the plots are not too informative. Maybe rescaling or a different way of presentation would help. Interpreting Figure 7/a is a bit difficult, probably a 3D plot would be useful to explain the different line sections.
assistant
This paper is trying to answer the question why ensembles of deep neural networks trained with random initialization work so well in practice in improving accuracy. Their proposed hypothesis is that networks trained from different initializations, although all converge to a low-loss/high accuracy optimum, explore different modes in function space and therefore provide more diversity. To experimentally support their hypothesis, first they show that functions along a single training trajectory are similar, however trajectories starting from different initializations may significantly differ. The difference in function space is based on the fraction of points on which the two functions disagree in terms of their prediction. Second, they use different subspace sampling methods around a single optimum and demonstrate that they are significantly less diverse (low disagreement between predictions) than sampling from independent optima through diversity vs accuracy plots. Moreover, they comment on the recent observation that local optima are connected by low-loss tunnels. They experimentally show that even though low-loss/high accuracy path exists between local optima, these tunnels do not correspond to similar solutions in function space, further supporting the multi-mode hypothesis. The authors compare the relative benefit of subspace sampling, weight averaging and ensembling on accuracy and interpret their findings in terms of the hypothesis. Overall, the paper is very well written and provides interesting insights into the multi-modal structure of deep neural network loss landscapes. Even though the hypothesis of the paper is not entirely new and has been touched upon in Fort & Jastrzebski (2019), this paper contributes to the field by providing thorough experimental support and clear exposition of the idea. Therefore, I would accept this paper if the authors provided additional experimental results on a different dataset. The paper mentions that the trends are consistent across all datasets the authors have explored. However, they only provide results on CIFAR-10 (and a limited set of experiments on ImageNet). Since the contribution of the paper heavily relies on providing experimental verification, it would be important to include at least the diversity vs. accuracy plot for the other datasets they have explored to demonstrate that this phenomenon is not specific to CIFAR-10. Additionally, I would like to add a couple of comments on the paper that are not part of my decision, but could potentially improve the paper. -The diversity score introduced in the paper is simple and intuitive, however it would be interesting to see whether the results hold across different notions of function space disagreement. -It is mentioned in the paper that data augmentation has been used for training the ResNet20 architecture. Would the results change significantly without data augmentation, as it adds another source of randomness to the training procedure. -Some comments on the figures: in Figure 3 it is very difficult to discern any difference between different shades of red (disagreement values), and in this form the plots are not too informative. Maybe rescaling or a different way of presentation would help. Interpreting Figure 7/a is a bit difficult, probably a 3D plot would be useful to explain the different line sections.