Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks. The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks. Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped). Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.
 I find myself having a very hard time making a review of this paper, because I mostly agree with the intro and discussion, and certainly agree that the "typical" versus "worse case" analysis is certainly an important point. The authors are making a strong case for the use of these models to understand overfitting and generalization in deep leaning. The problem is however that, except from advocating the use of these "spin glass" models studied back in the days by Seung, Sompolinksy, Opper and others, there are little new results presented in the paper. The arguments using the Very Simple Deep Learning (VSDL) are essentially a review of old known results --which I agree should maybe be revisited-- and the motivation to their application to deep learning stems from the reasoning that, since this is the behavior observed in all these model, well then deep learning should behave just the same as well. This might very well be, but this is precisly the point: is it ? After reading the paper, I agree with many points and enjoyed reading the discussion. I found interesting ideas discussed and many papers reviewed, and ended up discovering interesting papers on arxiv as a concequence. This is all nice, interesting, and well written, but at the end of the day, the paper is not doing too much beyond being a nice review of all ideas. While this has indeed some values, and might trigger a renewal of interested for these approaches, I will let the comity decide if this is the material they want in ICLR. A minor comment: The generalization result of [9,11] obtained with heuristic tools (the replica method of statistical mechanics) and plotted in Fig.1 (a) has been proven recently with rigorous mathematical methods in arxiv:1708.03395 Another remark: if deep learning is indeed well described by these models, then again so are many other simpler problems, such as compressed sensing, matrix and tensor factorization, error corrections, etc etc... with similar phase diagram as in fig. 1. For instance gaussian mixtures are discussed in http://iopscience.iop.org/article/10.1088/0305-4470/27/6/016/and SVM (which the authors argue should behave quite differently) methods have been treated by statistical mechanics tools in https://arxiv.org/pdf/cond-mat/9811421.pdf with similar phase diagrams. I am a bit confused what would be so special about deep learning then?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks. The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks. Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped). Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 I find myself having a very hard time making a review of this paper, because I mostly agree with the intro and discussion, and certainly agree that the "typical" versus "worse case" analysis is certainly an important point. The authors are making a strong case for the use of these models to understand overfitting and generalization in deep leaning. The problem is however that, except from advocating the use of these "spin glass" models studied back in the days by Seung, Sompolinksy, Opper and others, there are little new results presented in the paper. The arguments using the Very Simple Deep Learning (VSDL) are essentially a review of old known results --which I agree should maybe be revisited-- and the motivation to their application to deep learning stems from the reasoning that, since this is the behavior observed in all these model, well then deep learning should behave just the same as well. This might very well be, but this is precisly the point: is it ? After reading the paper, I agree with many points and enjoyed reading the discussion. I found interesting ideas discussed and many papers reviewed, and ended up discovering interesting papers on arxiv as a concequence. This is all nice, interesting, and well written, but at the end of the day, the paper is not doing too much beyond being a nice review of all ideas. While this has indeed some values, and might trigger a renewal of interested for these approaches, I will let the comity decide if this is the material they want in ICLR. A minor comment: The generalization result of [9,11] obtained with heuristic tools (the replica method of statistical mechanics) and plotted in Fig.1 (a) has been proven recently with rigorous mathematical methods in arxiv:1708.03395 Another remark: if deep learning is indeed well described by these models, then again so are many other simpler problems, such as compressed sensing, matrix and tensor factorization, error corrections, etc etc... with similar phase diagram as in fig. 1. For instance gaussian mixtures are discussed in http://iopscience.iop.org/article/10.1088/0305-4470/27/6/016/and SVM (which the authors argue should behave quite differently) methods have been treated by statistical mechanics tools in https://arxiv.org/pdf/cond-mat/9811421.pdf with similar phase diagrams. I am a bit confused what would be so special about deep learning then?
assistant
 I find myself having a very hard time making a review of this paper, because I mostly agree with the intro and discussion, and certainly agree that the "typical" versus "worse case" analysis is certainly an important point. The authors are making a strong case for the use of these models to understand overfitting and generalization in deep leaning. The problem is however that, except from advocating the use of these "spin glass" models studied back in the days by Seung, Sompolinksy, Opper and others, there are little new results presented in the paper. The arguments using the Very Simple Deep Learning (VSDL) are essentially a review of old known results --which I agree should maybe be revisited-- and the motivation to their application to deep learning stems from the reasoning that, since this is the behavior observed in all these model, well then deep learning should behave just the same as well. This might very well be, but this is precisly the point: is it ? After reading the paper, I agree with many points and enjoyed reading the discussion. I found interesting ideas discussed and many papers reviewed, and ended up discovering interesting papers on arxiv as a concequence. This is all nice, interesting, and well written, but at the end of the day, the paper is not doing too much beyond being a nice review of all ideas. While this has indeed some values, and might trigger a renewal of interested for these approaches, I will let the comity decide if this is the material they want in ICLR. A minor comment: The generalization result of [9,11] obtained with heuristic tools (the replica method of statistical mechanics) and plotted in Fig.1 (a) has been proven recently with rigorous mathematical methods in arxiv:1708.03395 Another remark: if deep learning is indeed well described by these models, then again so are many other simpler problems, such as compressed sensing, matrix and tensor factorization, error corrections, etc etc... with similar phase diagram as in fig. 1. For instance gaussian mixtures are discussed in http://iopscience.iop.org/article/10.1088/0305-4470/27/6/016/and SVM (which the authors argue should behave quite differently) methods have been treated by statistical mechanics tools in https://arxiv.org/pdf/cond-mat/9811421.pdf with similar phase diagrams. I am a bit confused what would be so special about deep learning then?