Learning deep neural networks could be understood as the combination of representation learning and learning halfspaces. While most previous work aims to diversify representation learning by data augmentations and regularizations, we explore the opposite direction through the lens of empirical Bayes method. Specifically, we propose a matrix-variate normal prior whose covariance matrix has a Kronecker product structure to capture the correlations in learning different neurons through backpropagation. The prior encourages neurons to learn from the experience of others, hence it provides an effective regularization when training large networks on small datasets. To optimize the model, we design an efficient block coordinate descent algorithm with analytic solutions. Empirically, we show that the proposed method helps the network converge to better local optima that also generalize better, and we verify the effectiveness of the approach on both multiclass classification and multitask regression problems with various network structures.
Summary: The submission proposes a method to learn Kronecker factors of the covariance of a matrix-variate normal distribution over neural network weights. Given a setting of the neural network weights, the Kronecker factors can be found in closed form by solving a convex optimization problem. 
Strengths:
+ The paper provides a thorough theoretical motivation for computing a positive definite factorization of the regularization term in the empirical Bayes setup.
Weaknesses:
- The novelty of the method is overstated.
- The method is claimed to be efficient, but each iteration requires an inner loop of solving a MAP problem (via gradient descent on the negative log likelihood with an extra regularization term), which is at least as expensive as a standard training run.
- The submission lacks precise technical writing, and many technical details appear in inappropriate places, such as the introduction.
- The experimental evaluation is not strong.
Major comments:
- The use of empirical Bayes is not novel in the context of neural networks despite the submission's claim that "Existing studies on
parametric empirical Bayes methods focus on the setting where the likelihood function and the prior are assumed to have specific forms" (pg. 1). In particular, see e.g., https://papers.nips.cc/paper/6864-an-empirical-bayes-approach-to-optimizing-machine-learning-algorithms.pdf, https://arxiv.org/abs/1801.08930, https://arxiv.org/abs/1807.01613 for the use of non-conjugate likelihoods in empirical Bayes.
- The cited motivation for the use of a matrix-variate normal prior over the weights of a neural network is weak. In particular, one iteration of credit assignment via backpropagation in a one-layer neural network does not adequately describe the complex interactions between parameters of a nonlinear model over the course of an optimization procedure such as the one used in line 2 of Algorithm 1. In addition, a learned prior can be used to introduce additional correlations between parameters, so it is strange to describe the learned prior as "capturing correlations" resulting from a single weight update.
- The submission lacks clarity on the assumptions entailed by using the proposed methodology, namely that the Kronecker factors are assumed positive definite. For instance, the logdet function is defined for positive definite (PD) matrices, the results in the paragraph titled "Approximate Volume Minimization" hold only for PD matrices, and the InvThresholding procedure is valid for only the same class. It should additionally be noted that this assumption is *not* required for a Kronecker factorization to be defined.
- The submission makes heavy use of Kronecker factorization, but neglects to cite works that use a similar factorization of the covariance matrices for neural network applications (e.g., https://arxiv.org/abs/1503.05671, https://arxiv.org/abs/1712.02390). Furthermore, the method bears a strong similarity to https://arxiv.org/abs/1506.02117 in learning a Kronecker-factored covariance structure between parameters of a neural network. Can the authors comment on the similarities and differences?
- Results are reported on a simple regression task (SARCOS) and multiclass classification problems (MNIST & CIFAR10) using a neural network with a single hidden layer. Moreover, "in all the experiments, the AEB algorithm is performed on the softmax layer" (pg. 7) and the justification for this in the "Ablations" section is opaque to me. Was a similar restriction used for L2 weight decay regularization? I can't interpret how thorough the evaluation is without such details. It is also not clear that the approach is extensible to more complex architectures, or that there would be a significant empirical benefit if this is done. 
- Figure 5 does not really exhibit interesting learned structure in the correlation matrix. Why not plot a visualization of the learned prior, rather than the weights?
Minor comments:
- The submission needs to be checked for English grammar and style.
- abstract: "Learning deep neural networks could be understood as the combination of representation learning and learning halfspaces." This is unclear.
- pg. 2: "Empirically, we show that the proposed method helps the network
converge to better local optima that also generalize better..."  What is a "better" local optimum?
- Section 3.1 describes empirical Bayes with point estimates. Please make it clearer that this methodology is not itself a contribution of paper by citing prior work.
- pg. 5: "Alg. 1 terminates when a stationary point is found." What exactly is the stopping criterion?
- pg. 6: The labels in Figure 2 are extremely small. Moreover, please keep the y-axis range constant.
- pg. 6, Figure 2 caption: "AEB improves generalization under both minibatch settings and is most beneficial when training set is small." Do the CIFAR10 results not show the opposite effect, that the regularization is most beneficial when the training set is large?
- pg. 7: "Batch Normalization suffers from large batch size in CIFAR10" weird wording
- pg. 8: "One future direction is to develop a better approximate solution to optimize the two covariance matrices from the marginal log-likelihood function." This is unclear.