On PAC-Bayes Bounds for Deep Neural Networks using the Loss Curvature
We investigate whether it*s possible to tighten PAC-Bayes bounds for deep neural networks by utilizing the Hessian of the training loss at the minimum. For the case of Gaussian priors and posteriors we introduce a Hessian-based method to obtain tighter PAC-Bayes bounds that relies on closed form solutions of layerwise subproblems. We thus avoid commonly used variational inference techniques which can be difficult to implement and time consuming for modern deep architectures. We conduct a theoretical analysis that links the random initialization, minimum, and curvature at the minimum of a deep neural network to limits on what is provable about generalization through PAC-Bayes. Through careful experiments we validate our theoretical predictions and analyze the influence of the prior mean, prior covariance, posterior mean and posterior covariance on obtaining tighter bounds.
Summary: The paper provides several approximations of PAC-Bayes generalization bounds for Gaussian prior and posterior distributions, with various restrictions on the covariance matrices. In particular, the paper: (1) Assumes that the expectation of the loss can be Taylor expanded around each point in the support, and all but the quadratic (Hessian) term can be ignored. (2) Proves a lower bound on the PAC-Bayes generalization objective. (3) Provides an upper bound on the PAC-Bayes objective via a *layerwise* Hessian objective. Evaluation: I found this paper extremely difficult to follow because it*s sloppy in various places -- both in terms of what claims are formal, and what are heuristic approximations -- and in terms of properly defining crucial quantities. I will go in the same numbered order in which I listed the main contributions above: (1) (Taylor approximation): The equation (4) is an *approximation* -- not a lower or upper bound. Moreover, too little is said about this heuristic: note that the authors actually Taylor expand an *expectation* over 	heta -- the trivial thing to require for this Taylor approximation to hold is that it holds for *every* theta which clearly will not be true. It seems the authors want to say the distribution Q concentrates over thetas close to some local optimum 	heta^*, and over these thetas the approximation holds. At the very least something needs to be said about how much things need to concentrate and whether this is realistic in real-life settings. Also, because (4) is an approximation, it*s a little disengenuous to call Theorems 4.2 and 5.2 *theorems*, and it needs to be mentioned in the statements that they hold under some formalization of the approximation I described above. (2) The lower bound is written very oddly -- the *prior* for the lower bound is really dependent upon the posterior -- so it is very strange to call it an *invalid* prior. Moreover, I have serious problems evaluating the meaning of this lower bound -- as it uses the Taylor approximation from (1), but then decides to instantiate the prior *depending on the optimum* of this Taylor approximation. As such, *at the very least* -- some small neural net examples should be tried where the normal (un-approximated) KL bound can be evaluated, to check whether this *actually* is a lower bound most of the time. (3) The upper bound is also written rather sloppily: Q_{lj} is never defined; H_{lj} only depends on l, rather than j -- in fact, I*m fairly sure it should be H_l, and eta should be sampled from Q_{l} (i.e. a vector with a coordinate for each neuron in layer l) if I understood the proof correctly.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On PAC-Bayes Bounds for Deep Neural Networks using the Loss Curvature
We investigate whether it*s possible to tighten PAC-Bayes bounds for deep neural networks by utilizing the Hessian of the training loss at the minimum. For the case of Gaussian priors and posteriors we introduce a Hessian-based method to obtain tighter PAC-Bayes bounds that relies on closed form solutions of layerwise subproblems. We thus avoid commonly used variational inference techniques which can be difficult to implement and time consuming for modern deep architectures. We conduct a theoretical analysis that links the random initialization, minimum, and curvature at the minimum of a deep neural network to limits on what is provable about generalization through PAC-Bayes. Through careful experiments we validate our theoretical predictions and analyze the influence of the prior mean, prior covariance, posterior mean and posterior covariance on obtaining tighter bounds.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Summary: The paper provides several approximations of PAC-Bayes generalization bounds for Gaussian prior and posterior distributions, with various restrictions on the covariance matrices. In particular, the paper: (1) Assumes that the expectation of the loss can be Taylor expanded around each point in the support, and all but the quadratic (Hessian) term can be ignored. (2) Proves a lower bound on the PAC-Bayes generalization objective. (3) Provides an upper bound on the PAC-Bayes objective via a *layerwise* Hessian objective. Evaluation: I found this paper extremely difficult to follow because it*s sloppy in various places -- both in terms of what claims are formal, and what are heuristic approximations -- and in terms of properly defining crucial quantities. I will go in the same numbered order in which I listed the main contributions above: (1) (Taylor approximation): The equation (4) is an *approximation* -- not a lower or upper bound. Moreover, too little is said about this heuristic: note that the authors actually Taylor expand an *expectation* over 	heta -- the trivial thing to require for this Taylor approximation to hold is that it holds for *every* theta which clearly will not be true. It seems the authors want to say the distribution Q concentrates over thetas close to some local optimum 	heta^*, and over these thetas the approximation holds. At the very least something needs to be said about how much things need to concentrate and whether this is realistic in real-life settings. Also, because (4) is an approximation, it*s a little disengenuous to call Theorems 4.2 and 5.2 *theorems*, and it needs to be mentioned in the statements that they hold under some formalization of the approximation I described above. (2) The lower bound is written very oddly -- the *prior* for the lower bound is really dependent upon the posterior -- so it is very strange to call it an *invalid* prior. Moreover, I have serious problems evaluating the meaning of this lower bound -- as it uses the Taylor approximation from (1), but then decides to instantiate the prior *depending on the optimum* of this Taylor approximation. As such, *at the very least* -- some small neural net examples should be tried where the normal (un-approximated) KL bound can be evaluated, to check whether this *actually* is a lower bound most of the time. (3) The upper bound is also written rather sloppily: Q_{lj} is never defined; H_{lj} only depends on l, rather than j -- in fact, I*m fairly sure it should be H_l, and eta should be sampled from Q_{l} (i.e. a vector with a coordinate for each neuron in layer l) if I understood the proof correctly.
assistant
Summary: The paper provides several approximations of PAC-Bayes generalization bounds for Gaussian prior and posterior distributions, with various restrictions on the covariance matrices. In particular, the paper: (1) Assumes that the expectation of the loss can be Taylor expanded around each point in the support, and all but the quadratic (Hessian) term can be ignored. (2) Proves a lower bound on the PAC-Bayes generalization objective. (3) Provides an upper bound on the PAC-Bayes objective via a *layerwise* Hessian objective. Evaluation: I found this paper extremely difficult to follow because it*s sloppy in various places -- both in terms of what claims are formal, and what are heuristic approximations -- and in terms of properly defining crucial quantities. I will go in the same numbered order in which I listed the main contributions above: (1) (Taylor approximation): The equation (4) is an *approximation* -- not a lower or upper bound. Moreover, too little is said about this heuristic: note that the authors actually Taylor expand an *expectation* over 	heta -- the trivial thing to require for this Taylor approximation to hold is that it holds for *every* theta which clearly will not be true. It seems the authors want to say the distribution Q concentrates over thetas close to some local optimum 	heta^*, and over these thetas the approximation holds. At the very least something needs to be said about how much things need to concentrate and whether this is realistic in real-life settings. Also, because (4) is an approximation, it*s a little disengenuous to call Theorems 4.2 and 5.2 *theorems*, and it needs to be mentioned in the statements that they hold under some formalization of the approximation I described above. (2) The lower bound is written very oddly -- the *prior* for the lower bound is really dependent upon the posterior -- so it is very strange to call it an *invalid* prior. Moreover, I have serious problems evaluating the meaning of this lower bound -- as it uses the Taylor approximation from (1), but then decides to instantiate the prior *depending on the optimum* of this Taylor approximation. As such, *at the very least* -- some small neural net examples should be tried where the normal (un-approximated) KL bound can be evaluated, to check whether this *actually* is a lower bound most of the time. (3) The upper bound is also written rather sloppily: Q_{lj} is never defined; H_{lj} only depends on l, rather than j -- in fact, I*m fairly sure it should be H_l, and eta should be sampled from Q_{l} (i.e. a vector with a coordinate for each neuron in layer l) if I understood the proof correctly.