Representing Model Uncertainty of Neural Networks in Sparse Information Form
This paper addresses the problem of representing a system*s belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN). The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs. To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form. Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis. As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank approximation of this eigenbasis and a memory-efficient sampling scheme. We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.
The contribution of the paper is marginal, as the principle of imposing Gaussians on the network to perform Bayes is not new. The Laplace-based approximation is half-baked and certainly much better techniques for Bayes exist in the recent literature; furthermore, it*s selection is not substantiated enough, and, of course, it represents no novelty. The experimental results are not convincing, as both the considered scenarios are limited and the comparisons are too poor (no consideration of state of the art alternatives). The provided corollaries are not actually helpful and should be put in the appendix.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Representing Model Uncertainty of Neural Networks in Sparse Information Form
This paper addresses the problem of representing a system*s belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN). The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs. To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form. Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis. As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank approximation of this eigenbasis and a memory-efficient sampling scheme. We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The contribution of the paper is marginal, as the principle of imposing Gaussians on the network to perform Bayes is not new. The Laplace-based approximation is half-baked and certainly much better techniques for Bayes exist in the recent literature; furthermore, it*s selection is not substantiated enough, and, of course, it represents no novelty. The experimental results are not convincing, as both the considered scenarios are limited and the comparisons are too poor (no consideration of state of the art alternatives). The provided corollaries are not actually helpful and should be put in the appendix.
assistant
The contribution of the paper is marginal, as the principle of imposing Gaussians on the network to perform Bayes is not new. The Laplace-based approximation is half-baked and certainly much better techniques for Bayes exist in the recent literature; furthermore, it*s selection is not substantiated enough, and, of course, it represents no novelty. The experimental results are not convincing, as both the considered scenarios are limited and the comparisons are too poor (no consideration of state of the art alternatives). The provided corollaries are not actually helpful and should be put in the appendix.