Variational Bayesian Inference is a popular methodology for approximating posterior distributions in Bayesian neural networks. Recent work developing this class of methods has explored ever richer parameterizations of the approximate posterior in the hope of improving performance. In contrast, here we share a curious experimental finding that suggests instead restricting the variational distribution to a more compact parameterization. For a variety of deep Bayesian neural networks trained using Gaussian mean-field variational inference, we find that the posterior standard deviations consistently exhibit strong low-rank structure after convergence. This means that by decomposing these variational parameters into a low-rank factorization, we can make our variational approximation more compact without decreasing the models' performance. Furthermore, we find that such factorized parameterizations improve the signal-to-noise ratio of stochastic gradient estimates of the variational lower bound, resulting in faster convergence.
The paper considers variational Bayesian inference (learning) for neural networks assuming that both the prior distributions and the posterior distributions of the network weights are factorising Gaussians. It is well known that the respective optimisation task for the parameters of the posterior weight distributions (ELBO) is tractable by stochastic gradient ascent.  The authors propose to simplify (restrict) the model even further, by assuming that the posterior variances for fully connected layers, (when seen as a matrix) have low rank. It is pretty obvious that the ELBO optimisation task remains tractable in this case.
The authors perform two types of experiments to show that the proposed simpler model does not decrease the performance of the network, when compared to the full rank factorising model. They first show that the learned variances of dense layers indeed exhibit a low rank structure for three models learned on the corresponding data (MLP on MNIST, LeNet on CIFAR and LSTM on IMDB). Then, in a second step, they consider the proposed rank constraint during learning and show that with rank k >= 2 the models are able to achieve performance competitive with the full rank model and, moreover, exhibit better signal to noise ratio for the gradients during learning.
The paper is well written, clearly structured and technically correct. However, in my opinion, its novelty and new insights are restricted, which is why I suggest to reject the paper. The reasons for this are the following.
- The authors restrict their analysis to dense layers only. Moreover, it remains conceptually unclear, why and when the proposed model should be useful and represent a good approximation of the full rank model. 
- The experimental study is restricted to small models, e.g. in the case of CIFAR a quite shallow version of LeNet which achieves only ~45% validation accuracy. It remains unclear, whether the observed low rank structure of the variance matrices (of the dense layers) will scale to deeper models that could achieve competitive validation accuracies.
- When measuring the impact of the low rank tying of the posterior variances, the authors compare to the full rank model only. I am missing the "point estimate" baseline for these models. For if the the positive impact of the Bayesian inference approach with the full rank model is small when compared to the point estimate, then, as a consequence, the impact of the low rank tying must be small when compared to the full rank model.
- The numbers reported in Table 3 (second group of experiments) raise some questions not addressed by the authors. It remains unclear, why the low rank model with ranks 1,2,3 gives better accuracies on CIFAR than the full rank model. The same holds for the LSTM experiment. This may indicate some kind of "overfitting" and should have been analysed by the authors in order to "disentangle" possible overfitting issues from the question of validity of the proposed low rank model.