This paper presents a sparse representation of model uncertainty for deep neural networks (DNNs) that relies on an inverse formulation of Multivariate Normal Distribution (MND): an information form. We show that the model uncertainty can be estimated in this form using a scalable Laplace Approximation scheme, which involves a diagonal correction of the Kronecker-factored eigenbasis. As this makes the inversion of the information matrix intractable -an operation that is required for a full Bayesian analysis, we further devise a novel low-rank approximation of this eigenbasis that exploits spectral sparsity of DNNs. Methods to realize this sparsification are provided that develops into a memory-wise tractable sampling computations. Both of our theoretical analysis and empirical evaluations over various benchmarks show the superiority of our approach over existing methods.
Laplace approximation has been an important tool for obtaining uncertainty estimation for deterministic models. To efficiently approximate the Hessian matrix for neural networks, Ritter et. al (2018) proposes to use K-FAC . Motivated by the relatively inaccurate approximations of K-FAC, this paper proposes to improve KFAC approximations by combining eigen-basis corrections (EK-FAC, George et al, 2018) and diagonal corrections. The paper shows that the proposed method has smaller Frobenius approximate errors compared to K-FAC and EK-FAC. To further reduce the computational costs, the paper proposes a low-rank approximation by keeping only the L largest eigenvalues. Empirically, the paper demonstrates improved calibration and out-of-distribution entropies compared to previous approaches. 
# Diagonal Corrections. 
With the diagonal correction approximating the Fisher better, the paper shows the computation can still be conducted in the scale of W, which is similar to K-FAC. However, the method requires the diagonal correction matrix D is always positive, which might not be true. Moreover, because the computation requires D^{-1}, clipping D to a small constant will bring up stability issues. I wonder how this problem is tackled in this paper. 
# Writing 
The paper's notations are messy, which requires a lot of intellectual guesses to understand the conveyed idea. 
1) Notations are not introduced, such as $\delta \theta$, $W_{map}^{IV}$, the MN distribution in the appendix.
2) Notations are typoed. Eq(3) $N(0, A^{-1} \otimes G^{-1}) = MN(0, G^{-1}, A^{-1})$; The bottom paragraph in Page 4; eq(8).
3) Notations are abused. In particular for $V$ and $\Lambda$ when introducing EK-FAC. The paper uses $V$ for both true eigenbasis and the EK-FAC eigenbasis. In addition, the EK-FAC part should be moved to the background section.
4) The paragraph below Corollary 1 says the author can prove the proposed method also has closer approximations in terms of the Fisher inverse. But no proofs are given.
5) Caption of Figure4.
# Laplace Approximation 
For eq(4, 8, 12), The Hessian in Laplace approximations should be divided by $N$. Although the paper also mentions the scaling below those equations. But technically eq(4, 8, 12) are wrong and I don't know whether the experiments really did the scaling or not. 
# Low-rank Approximation
1) It is not clear why the low-rank approximation is necessary. The computational costs are inevitable to compute the eigen-system of A and G. Why do we need the low-rank approximation after that ? K-FAC is not a computationally expensive method either. 
2) I cannot understand the proofs of Lemma 2. In fact, I don't know what $ I_{1:L}^{top}, I_{1:K}^{top}$ means. More explicit formulas should be given for clarity. 
3) Lemma 4 states $I_ii = (\hat{I}_{def})_ii$. Although the diagonal correction makes $I_ii = (I_{def})_ii$, the low-rank approximation makes them unequal again. Or I guess you use a different D in eq(13) from the D in eq(10) ? 
# Experiments 
The paper needs more experiments to validate the proposed method. Firstly, for the MNIST experiments, it is better to use the same architecture as in Ritter et al (2018) for direct comparisons. Beyond that, the adversarial attack experiment and the mis-classification uncertainty experiment in Ritter et al (2018) seem to be good choices as well. 
# Overall
The paper proposes a diagonal corrected EK-FAC, achieving better approximation of the Fisher matrix. Interestingly, the paper shows that this corrections doesn't add too much computations.  However, the proposed low-rank approximation doesn't seem necessary. And the paper's notations and presentations are too messy to be an accepted paper.