While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly.
This paper gives various PAC-Bayesian generalization guarantees and some
empirical results on parameter perturbation in training using an algorithm
motivated by the theory.
The fundamental issue addressed in this paper is whether parameter
perturbation during training improves generalization and, if so, what
theoretical basis exists for this phenomenon.  For continuously
parameterized models, PAC-Bayesian bounds are fundamentally based on
parameter perturbation (non-singular posteriors).  So PAC-Bayesian
theory is naturally tied to parameter perturbation issues.  A more
refined question is whether the size of the perturbation should be
done on a per-parameter bases and whether per-parameter noise levels
should be adaptive --- should the appropriate noise level for each
parameter be adjusted on the basis of statistics in the training data.
Adam and RMS-prop both adapt per-parameter learning rate eta_i to be
proportional to 1/((E g_i^2) + epsilon) where E g_i^2 is some running
estimate of the expectation over a draw of a training point of the
square of the gradient of the loss with respect to parameter i.  At
the end of the day, this paper, based on PAC-Bayesian analysis,
proposes that a very similar adaptation be made to per-parameter noise
during training but where E g_i^2 is replaced by the RMS value \sqrt{E
g_i^2}.  It seems that all theoretical analyses require the square
root --- the units need to work.  A fundamental theoretical question,
perhaps unrelated to this paper, is why in learning rate adaptation the
square root hurts the performance.
This paper can be evaluated on both theoretical and empirical grounds.
At a theoretical level I have several complaints.  First, the
theoretical analysis seem fairly mechanical and without theoretical
innovation. Second, the analysis obscures the prior being used (the
learning bias). The paper first states an assumption that each
parameter is a-priori taken to be uniform over |w_i| <= \tau_i and the
KL-divergence in the PAC-Bayes bound is then log tau_i/sigma_i where
sigma_i is the width of a uniform posterior over a smaller interval.
But later they say that they approximate tau_i by |w_i| + kappa_i with
kappa_i = \gamma |w_i| + epsilon.  I believe this works out to be
essentially a log-uniform prior on |w_i| (over some finite range of
log |w_i|).  This seems quite reasonable but should be made explicit.
The paper ignores the possibility that the prior should be centered at
the random initialization of the parameters.  This was found to be
essential in Dziugaite and Roy and completely changes the dependence
of k_i on w_i.
Another complaint is that the Hoefding bound is very loose in cases
where the emperical loss is small compared to its upper bound.  The
analysis can be more intuitively related to practice by avoiding the
rescaling of the loss into the interval [0,1] and writing expressions
in terms of a maximum bound on the loss L_max.  When hat{L} << L_max
(almost always the case in practice) the relative Chernoff bound is
much tighter and significantly alters the analysis.  See McAllester's
PAC-Bayesian tutorial.
The theoretical discussion on re-parameterization misses an important
point, in my opinoin, relative to the need to impose a learning bias
(the no-free-lunch theorem).  All L_2 generalization bounds can be
interpreted in terms of a Gaussian prior on the parameters.  In all
such cases the prior (the learning bias) is not invariant to
re-parameterization.  All L_2 generalization bounds are subject to the
same re-parameterization criticism.  A prior tied to a particular
parameterization is standard practice in machine learning for in all
L_2 generalization bounds, including SVMs.  I do think that a
log-uniform prior (rather than a Gaussian prior) is superior and
greatly reduces sensitivity to re-parameterization as noted by the
authors (extremely indirectly).
I did not find the empirical results to very useful.  The value of
parameter perturbation in training remains an open question. Although
it is rarely done in practice today, it is an important fundamental
question. A much more thorough investigation is needed before any
conclusions can be drawn with confidence. Experimentation with
perturbation methods would seem more informative than theory given the
current state of the art in relating theory to practice.