Representing Model Uncertainty of Neural Networks in Sparse Information Form
This paper addresses the problem of representing a system*s belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN). The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs. To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form. Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis. As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank approximation of this eigenbasis and a memory-efficient sampling scheme. We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.
The submitted paper presents a method of approximating the posterior distribution over the DNN parameters based on a Laplace Approximation scheme. It extends on the previous work by adding a diagonal correction term to the Kronecker-factored eigenbasis and also suggests a low-rank representation of Kronecker-factored eigendecomposition. Empirical evaluations were done to show that the proposed method has a more accurate uncertainty estimation compared to the previous work. Overall, the paper is well-organized and easy to follow, although mathematical notations are not consistent throughout the paper. It is well-referenced, and the derivations look mostly correct (explained in minor comments). The main idea of the paper is convincing and well-motivated. To my knowledge, the proposed method of adding a correction term has not been introduced before. However, it is more of an incremental contribution to the existing works. In that sense, I am slightly concerned that its novelty is limited. The experiments are not comprehensive. For the toy regression problem, a comparison to Hamiltonian Monte Carlo would be more informative. Moreover, it would be helpful to report the comparison with factorized variational methods (e.g. Graves, 2011) and experiment on modern architectures. I would be also interested to see what the additional time complexity by adding a diagonal correction term is and how more efficient it gets by low-rank approximation in experimental details. Minor comments: *Page 4: It would have been easier to understand if there had been a notational distinction between exact eigenbasis and K-FAC eigenbasis in defining V. Also, “In equation equation 10” -> “In equation 10”. *Page 7: In Lemma 2, the order in describing the low-rank estimate does not appear to be correct. Also, in Lemma 4, shouldn’t there be a hat on I_{efb} and I_{kfac}? *Page 9: The colour scheme in Figure 3 looks visually harder to read. *Page 14: In equation 15 and 16, there should be a bracket for (2 pi), and in Appendix B, some variables (e.g. p, k, R) are left unexplained. *Page 14, 15: In equation 23 and proposition 1, mathcal is missing, and vec operator is missing in equation 22. *Page 3: In equation 2 and 4, is it A_{i-1} or A_i? *Page 16: In equation 27, the dimension of X in mathcal{R}^{m 	imes m} seems incorrect. Also, in the last line, doesn’t X odot D have different dimensions (similarly in equation 22)?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Representing Model Uncertainty of Neural Networks in Sparse Information Form
This paper addresses the problem of representing a system*s belief using multi-variate normal distributions (MND) where the underlying model is based on a deep neural network (DNN). The major challenge with DNNs is the computational complexity that is needed to obtain model uncertainty using MNDs. To achieve a scalable method, we propose a novel approach that expresses the parameter posterior in sparse information form. Our inference algorithm is based on a novel Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis. As this makes the inversion of the information matrix intractable - an operation that is required for full Bayesian analysis, we devise a low-rank approximation of this eigenbasis and a memory-efficient sampling scheme. We provide both a theoretical analysis and an empirical evaluation on various benchmark data sets, showing the superiority of our approach over existing methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The submitted paper presents a method of approximating the posterior distribution over the DNN parameters based on a Laplace Approximation scheme. It extends on the previous work by adding a diagonal correction term to the Kronecker-factored eigenbasis and also suggests a low-rank representation of Kronecker-factored eigendecomposition. Empirical evaluations were done to show that the proposed method has a more accurate uncertainty estimation compared to the previous work. Overall, the paper is well-organized and easy to follow, although mathematical notations are not consistent throughout the paper. It is well-referenced, and the derivations look mostly correct (explained in minor comments). The main idea of the paper is convincing and well-motivated. To my knowledge, the proposed method of adding a correction term has not been introduced before. However, it is more of an incremental contribution to the existing works. In that sense, I am slightly concerned that its novelty is limited. The experiments are not comprehensive. For the toy regression problem, a comparison to Hamiltonian Monte Carlo would be more informative. Moreover, it would be helpful to report the comparison with factorized variational methods (e.g. Graves, 2011) and experiment on modern architectures. I would be also interested to see what the additional time complexity by adding a diagonal correction term is and how more efficient it gets by low-rank approximation in experimental details. Minor comments: *Page 4: It would have been easier to understand if there had been a notational distinction between exact eigenbasis and K-FAC eigenbasis in defining V. Also, “In equation equation 10” -> “In equation 10”. *Page 7: In Lemma 2, the order in describing the low-rank estimate does not appear to be correct. Also, in Lemma 4, shouldn’t there be a hat on I_{efb} and I_{kfac}? *Page 9: The colour scheme in Figure 3 looks visually harder to read. *Page 14: In equation 15 and 16, there should be a bracket for (2 pi), and in Appendix B, some variables (e.g. p, k, R) are left unexplained. *Page 14, 15: In equation 23 and proposition 1, mathcal is missing, and vec operator is missing in equation 22. *Page 3: In equation 2 and 4, is it A_{i-1} or A_i? *Page 16: In equation 27, the dimension of X in mathcal{R}^{m 	imes m} seems incorrect. Also, in the last line, doesn’t X odot D have different dimensions (similarly in equation 22)?
assistant
The submitted paper presents a method of approximating the posterior distribution over the DNN parameters based on a Laplace Approximation scheme. It extends on the previous work by adding a diagonal correction term to the Kronecker-factored eigenbasis and also suggests a low-rank representation of Kronecker-factored eigendecomposition. Empirical evaluations were done to show that the proposed method has a more accurate uncertainty estimation compared to the previous work. Overall, the paper is well-organized and easy to follow, although mathematical notations are not consistent throughout the paper. It is well-referenced, and the derivations look mostly correct (explained in minor comments). The main idea of the paper is convincing and well-motivated. To my knowledge, the proposed method of adding a correction term has not been introduced before. However, it is more of an incremental contribution to the existing works. In that sense, I am slightly concerned that its novelty is limited. The experiments are not comprehensive. For the toy regression problem, a comparison to Hamiltonian Monte Carlo would be more informative. Moreover, it would be helpful to report the comparison with factorized variational methods (e.g. Graves, 2011) and experiment on modern architectures. I would be also interested to see what the additional time complexity by adding a diagonal correction term is and how more efficient it gets by low-rank approximation in experimental details. Minor comments: *Page 4: It would have been easier to understand if there had been a notational distinction between exact eigenbasis and K-FAC eigenbasis in defining V. Also, “In equation equation 10” -> “In equation 10”. *Page 7: In Lemma 2, the order in describing the low-rank estimate does not appear to be correct. Also, in Lemma 4, shouldn’t there be a hat on I_{efb} and I_{kfac}? *Page 9: The colour scheme in Figure 3 looks visually harder to read. *Page 14: In equation 15 and 16, there should be a bracket for (2 pi), and in Appendix B, some variables (e.g. p, k, R) are left unexplained. *Page 14, 15: In equation 23 and proposition 1, mathcal is missing, and vec operator is missing in equation 22. *Page 3: In equation 2 and 4, is it A_{i-1} or A_i? *Page 16: In equation 27, the dimension of X in mathcal{R}^{m 	imes m} seems incorrect. Also, in the last line, doesn’t X odot D have different dimensions (similarly in equation 22)?