We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks. The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks. Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped). Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.
The authors suggest that ideas from statistical mechanics will help to understand the "peculiar and counterintuitive generalization properties of deep neural networks." The paper's key claim (from the abstract) is that their approach "provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc." This claim is restated on p. 2, third full paragraph.
I am sympathetic to the idea that ideas from statistical mechanics are relevant to modern learning theory. However, I do not find this paper at all convincing. I find the paper incoherent: I am unable to understand the argument for the central claims. On the one hand, the paper seems to be written as a "response" to Zhang et al.'s "Understanding Deep Learning Requires Rethinking Generalization", (henceforth Z): the introduction mentions Z multiple times, and the title of this work refers to Z. On the other hand, none of the issues raised by Z are (as far as I can tell) addressed in any substantial way by this paper. In somewhat more detail, this work discusses two major observations:
1. Neural nets can easily overtrain, even to random data.
2. Popular ways to regularize may or may not help.
Z certainly observes 1 and arguably observes 2. (I'd argue against, see below, but it's at least arguable.) I do not see how this paper addresses either observation. Instead, what the statistical mechanics (SM) approach seems to do is explain (or predict) the existence of phase transitions, where we suddenly go from a regime of poor generalization to good generalization or vice versa. However, neither Z nor, as far as I can tell, any other reference given here, suggests that these phase transitions are frequently observed in modern deep learning. The most relevant bit from Z is Figure 1c, which suggests that as the noise level is increased (corresponding to alpha decreasing in this paper), the generalization error increases smoothly. This seems to be in direct contradiction to the predictions made by the theories presented here.
If the authors wish to hold to the claim that their work "can provide a qualitative explanation of recently-observed empirical properties that are not easily-understandable from within PAC/VC theory of generalization, as it is commonly-used in ML" (p. 2), it is absolutely critical that they be more specific about which specific observations from which papers they think they are explaining. As written, I simply do not see which actual observations they think they explain.
In observation 2, the authors suggest that many popular ways to implement regularization "do not substantially improve the situation". A careful reading of Z (and this was corroborated by discussion with the authors) is that Z observed that regularization with parameters commonly used in practice (or, put differently, regularization parameters that led to the highest holdout accuracy in other papers) still led to substantial overtraining on noisy data. I think it is almost certainly true (see below for more discussion) that much larger values of regularization can prevent overfitting, at the cost of underfitting. It's also worth noting that Z agrees with basically all practitioners that various regularization techniques can make an important difference to practitioners who want to minimize test error; what they don't do (at least at moderate values) is *qualitatively* destroy a network's ability to overfit to noise. It is unclear to me how this paper explains observation 2 (see below for extensive discussion).
I don't actually understand the first full paragraph on p. 2 well. It is true that we can always avoid overtraining by tuning regularization parameters to get better generalization *error* (difference beween train and test) on the test data set (but possibly worse generalization accuracy); the rest of the paper seems to take the opposite side on this. A Gaussian kernel SVM with a small enough bandwidth and small enough regularization parameter can also overfit to noise. The argument needs to be sharpened here.
I find the discussion of noise at the bottom of p. 2 confusing. The authors describe tau "having to do with noise in the learning process", but then suggest that "adding noise decreases the effective load." This is the first time noise is really talked about, and it seems like maybe noise in the data is about alpha, but noise in the "learning process" is about tau? This should be clarified.
On p. 3, the authors refer to "the two parameters used by Z and many others." I am honestly not sure what's being referred to here. I just reread Z and I don't get it.  What two parameters are used by Z?
p. 3, figure. The authors should be clear about what recent (ideally widely-discussed) experimental results look anything like this figure. I found nothing in Z et al. In Appendix A.4, there is a mention of Figure 3 of Chromanska et al. 2014; that figure also seems to be totally consistent with smooth transitions and does not (to me) present any obvious evidence of a sharp phase transition. (In any case, the main paper should not rely heavily on the appendix for its main empirical evidence.)
p. 3, figure 1a. What is essential in this figure? A single phase transition? That the error be very low on the r.h.s. of the phase transition (probably not that, judging from the related models in the
Appendix).
p. 3, figure 1b/c. What does SG stand for? As far as I can tell it's never discussed.
p. 4. "Thus, an important more general insight from our approach is that --- depending strongly on details of the model, the specific details of the learning algorithm, the detailed properties of the data and their noise etc. --- going beyond worst-case bounds can lead to a rich and complex array of manners in which generalization can depend on the control parameters of the ML process." This is well-known to all practitioners. This paper does not seem to offer any specific testable explanations or predictions of any sort. I certainly agree that the study of SM models is "interesting", but what would
make this valuable would be a more direct analogy, a direct explanation of some empirical phenomenon.
Section 2 in general. The authors discuss a couple different types of observations: (1) "strong discontinuities in generalization performance as a function of control parameters" aka phase transitions, and (2) generalization performance can depend sensitively on details of the model, details of algorithms, implicit regularization properties, detailed properties of data and noise, etc." (1) shows up in the SM literature from the 90's discussed in Appendix A. I don't think it shows up in modern practice, and I don't think it shows up in Z. (2) is absolutely relevant to modern practitioners, but I don't see what this paper has to say about it beyond "SM literature from the 90's exhibits similar phenomena." The model introduced in Section 3 abstracts all such concerns away.
Section 3. I am not super comfortable with the idea of "Claims", especially since the 3 Claims seem to be different sorts of things. I would normally think of a "Claim" as something that could be true or false, possibly with some argument for its truth.
Claim 1 introduces a model (VSDL), but I wouldn't call this a claim, since nothing is actually "claimed." The subpoints of Claim 1 are arguably claims, but they're not introduced as such. I address these
in turn:
"Adding noise decreases an effective load alpha." The paper states "N is the effective capacity of the model trained on these data", but "effective capacity" is never defined. Certainly, if we *define* alpha = m_eff / N and *define* m_eff = m - m_rand, the (sub)claim follows, but why are those definitions good?  I *think* what's going on here is hidden in the sentence "empirical results indicate that for realistic DNNs it is close to 1. Thus, the model capacity N of realistic DNNs scales with m and not m_eff.", where "it" refers to the Rademacher complexity. Well, OK, but if we agree with that, then aren't we just *assuming* the main result of Z rather than explaining it? We're basically just stating that the models can memorize the data?
I don't really understand the point the last part of the paragraph is trying to make (everything after what I quoted above).
"Early stopping increases an effective temperature tau." I find this plausible but don't understand the argument at all. To this reader, it's just "stuff from SM I don't understand." I think the typical ML reader of this paper won't necessarily be familiar with any of "the weights evolve according to a relaxation Langevin equation", "from the fluctuation-dissipation theorem", or the reference to annealing rate schedules. Consider either explaining this more or just appealing to SM and relegating this to an appendix.
After the claim, the paper mentions that the VSDL model ignores other "knobs". This is fine for a model, but I think it's totally disingenuous to then suggest that this model explains anything about other popular ways to regularize (Observation 2 in the intro, see also my comment on Section 2). In the intro, the claim is "Other regularizations sometimes help and sometimes don't and we don't understand why" (the claim is about overfitting but it's also true for improving performance in general), which is basically true. But introducing a model which completely abstracts these things away cannot possibly explain anything about the behavior.
Claim 2 is that we should consider a thermodynamic limit where model complexity grows with data (the paper says grows with the number of parameters, I assume this is a typo). I would probably call this one an "Assumption", with some arguments for the justification. I think this is one of the most interesting and important ideas in the paper, and I don't fully understand it, even after reading the appendix. I have questions. How should / could this apply to practitioners, who cannot in general hope to obtain arbitrary amounts of data? Are we assuming that any (or all) modern DNN experiments are in the asymptotic regime? Are we assuming the experiments in Z are in this regime? Is there any relevance to the fact that in an ML problem (unlike in say a spin glass, at least as far as I know) the "complexity" of the *task* is *not* increasing with the data size, so eventually one will have seen "enough" data to "saturate" the task?  I'd love to know more.
Claim 3 is more of an "Informal Theorem" that under the model of Claim 1 and the assumption of Claim 2, the phase diagrams of Figure 1 hold. The "proof" is a reference to SM papers. This should be clarified.
Yet again, I point out that I do not know any modern large-scale NN experiments that correspond to any of the pictures in Figure 1.
There's a mention of "tau = 0 or t > 0." What is the significance of tau = 0? How should an ML reader think about this?
Section 3.2 suggests that Claim 3 (the existence of the 1 and 2d phase diagrams) "explain" Observations 1 and 2 from the Appendix. I simply do not see this. 
For Observation 1, that NNs can easily overtrain, the "argument" seems to boil down to "the system is in a phase where it cannot help but overtrain." This is hardly an explanation at all. How do we know what phase these experiments were in? How do we know these experiments were in the thermodynamic limit?
For Observation 2, the authors point out that in VSDL, "the only way to prevent overfitting is to decrease the number of iterations." This seems true but vacuous: the authors introduced a model where regularization doesn't correspond to any knobs, so of course to the extent that that model explains reality, the knobs don't stop overfitting. But this feels like begging the question. If we accept the VSDL model, we'd also accept that various regularizations can't improve generalization, which goes directly against basically all practice. I guess I technically have to concede that "Given the three
claims", Observation 2 follows, but Claim 1 by itself seems to be already assuming the conclusion.
Minor writing issues:
The authors mention at least four times that reproducing others' results is not easy (p. 1 observation 1, p. 4 first paragraph, p. 4 footnote 6, last sentence of the main text). While I think this statement is true, it is quite well-known, and I suggest that the authors may simply alienate readers by harping on it here.
p. 1. "may always overtrain" is unclear. I don't know what it means. Is the claim that SOTA DNNs wll always overtrain when presented with enough data? I don't think so from the rest of the paper, but I'm not sure.
I'm a little unclear what the authors mean by "generalization error" (or "generalization accuracy", which seems to only be used on p. 2). Z use "generalization error = training error - test error". Check the appendix for consistency here too.
Replace "phenomenon" with "phenomena", at least twice where appropriate.
p. 3, first paragraph. I think the reference to the Hopfield model should be relegated to a footnote. The text "two or more such parameter holds more generally" is confusing; is it two, or is it two or more? What will I understand differently if I use more than two parameters? The next paragraph, starting with "Given these two identifications, which are novel to this work," seems odd, since we've
just seen 7+ references and a claim that they have similar parameterizations, so it's unclear what's novel.
Appendix A.5. "For non-linear dynamical systems... NNs from the 80s/90s or our VSDL model or realistic DNNs today .. there is simply no reason to expect this to be true." where "this" refers to "one can always choose a value of lambda to prevent overfitting, potentially at the expense of underfitting." I don't understand, and I also think this disagrees with the first full paragraph on p. 2. Is there some thermodynamic limit argument required here? The very next bullet states that x = argmin_x f(x) + lambda g(x) can prevent overfitting with large lambda. What's different? I'm overall not clear what's being implied here. Consider a modern DNN for classification. A network with all zero weights will have some empirical loss L(0). If I minimize, for the weights of a network w, L(w) + lambda ||w||^2, I have that L(w) + lambda ||w||^2 <= L(0) (assuming I can solve the optimization), and assuming L is non-negative, lambda ||w||^2 <= L(0), or ||w||^2 <= L(0) / lambda. So for very large lambda, I can drive ||w||^2 arbitrarily close to zero. How is this importantly different from the linear case?  What am I missing?
p. 3. "inability not to overfit." Avoid the double negative.
Intro, last paragraph. Weird section order description, with ref to Section A coming before section 4.
Footnote 2. "but it can be quite limiting." More detail needed. Limiting how?
Footnotes 3 and 4. The text says there are "technical" and "non-technical" reasons, but 3 and 4 both seem technical to me.
Appendix A.2. "on a randomly chosen subset of X." Is it really subset? Are we picking subsets uniformly at random?