Empirical Investigation on Model Capacity and Generalization of Neural Networks for Text
Recently, deep neural network models have shown promising opportunities for many natural language processing (NLP) tasks. In practice, the number of parameters of deep neural models is often significantly larger than the size of the training set, and its generalization behavior cannot be explained by the classic generalization theory. In this paper, with extensive experiments, we empirically investigate the model capacity and generalization of neural models for text. The experiments show that deep neural models can find patterns better than brute-force memorization. Therefore, a large-capacity model with early-stopping stochastic gradient descent (SGD) as implicit regularizer seems to be the best choice, as it has better generalization ability and higher convergence speed.
 The quality and clarity of this manuscript are somewhat low. While an empirical exploration of capacity and generalization on text could provide novel insights, the experiments are performed only on one task; a medium-small sentiment analysis dataset, and there are no novel conclusions reached. Pros: - experimental results are clearly presented - most details of experiments are explained - the basic outline of the paper explores an interesting question Cons - many sentences do not have a clear point, and there are many spelling and grammar errors - Despite claims to the contrary, the experiments are far from extensive; they explore two models, with variations in context for embeddings, with 3 optimizers, on just one task. To be an extensive discussion of "text", I would expect to see language-modeling and seq-to-seq tasks as well - the majority of the manuscript is taken up by overly long explanations of background concepts (e.g. half a page on risk minimization, half a page on "different kinds of neural networks" with vague and occasionally misleading descriptions of RNN vs CNN, containing zero citations) - the related work is significantly lacking, and it appears that the authors are unaware of results and published work which would be directly relevant to theirs (e.g. Arpit et al. A closer look at memorization in deep networks ; Advani and Saxe Generalization error dynamics in high dimensions) - in the introduction, generalization is not clearly explained, which seems kind of like an important thing to do... - VC is a measure/set, not a norm? - the section "Linked to the mode selection of distribution" is unclear. I don*t see the justification for assuming that a decrease in test accuracy indicates mode selection, and don*t understand figure 4b (it*s just two bumps labelled mode 1 and mode 2 with an arrow between them. What are the axes of this plot?? Is this real data, or just a vague illustration of mode selection?) - Did you tune the learning rate for the adagrad and gradient descent methods? I find it very surprising that they would fail to converge. - experiments are not described in sufficient detail to be reproduced. What are the other hyperparameters for the various experiments? Only test set results are plotted; what about training/validation curves?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Empirical Investigation on Model Capacity and Generalization of Neural Networks for Text
Recently, deep neural network models have shown promising opportunities for many natural language processing (NLP) tasks. In practice, the number of parameters of deep neural models is often significantly larger than the size of the training set, and its generalization behavior cannot be explained by the classic generalization theory. In this paper, with extensive experiments, we empirically investigate the model capacity and generalization of neural models for text. The experiments show that deep neural models can find patterns better than brute-force memorization. Therefore, a large-capacity model with early-stopping stochastic gradient descent (SGD) as implicit regularizer seems to be the best choice, as it has better generalization ability and higher convergence speed.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The quality and clarity of this manuscript are somewhat low. While an empirical exploration of capacity and generalization on text could provide novel insights, the experiments are performed only on one task; a medium-small sentiment analysis dataset, and there are no novel conclusions reached. Pros: - experimental results are clearly presented - most details of experiments are explained - the basic outline of the paper explores an interesting question Cons - many sentences do not have a clear point, and there are many spelling and grammar errors - Despite claims to the contrary, the experiments are far from extensive; they explore two models, with variations in context for embeddings, with 3 optimizers, on just one task. To be an extensive discussion of "text", I would expect to see language-modeling and seq-to-seq tasks as well - the majority of the manuscript is taken up by overly long explanations of background concepts (e.g. half a page on risk minimization, half a page on "different kinds of neural networks" with vague and occasionally misleading descriptions of RNN vs CNN, containing zero citations) - the related work is significantly lacking, and it appears that the authors are unaware of results and published work which would be directly relevant to theirs (e.g. Arpit et al. A closer look at memorization in deep networks ; Advani and Saxe Generalization error dynamics in high dimensions) - in the introduction, generalization is not clearly explained, which seems kind of like an important thing to do... - VC is a measure/set, not a norm? - the section "Linked to the mode selection of distribution" is unclear. I don*t see the justification for assuming that a decrease in test accuracy indicates mode selection, and don*t understand figure 4b (it*s just two bumps labelled mode 1 and mode 2 with an arrow between them. What are the axes of this plot?? Is this real data, or just a vague illustration of mode selection?) - Did you tune the learning rate for the adagrad and gradient descent methods? I find it very surprising that they would fail to converge. - experiments are not described in sufficient detail to be reproduced. What are the other hyperparameters for the various experiments? Only test set results are plotted; what about training/validation curves?
assistant
 The quality and clarity of this manuscript are somewhat low. While an empirical exploration of capacity and generalization on text could provide novel insights, the experiments are performed only on one task; a medium-small sentiment analysis dataset, and there are no novel conclusions reached. Pros: - experimental results are clearly presented - most details of experiments are explained - the basic outline of the paper explores an interesting question Cons - many sentences do not have a clear point, and there are many spelling and grammar errors - Despite claims to the contrary, the experiments are far from extensive; they explore two models, with variations in context for embeddings, with 3 optimizers, on just one task. To be an extensive discussion of "text", I would expect to see language-modeling and seq-to-seq tasks as well - the majority of the manuscript is taken up by overly long explanations of background concepts (e.g. half a page on risk minimization, half a page on "different kinds of neural networks" with vague and occasionally misleading descriptions of RNN vs CNN, containing zero citations) - the related work is significantly lacking, and it appears that the authors are unaware of results and published work which would be directly relevant to theirs (e.g. Arpit et al. A closer look at memorization in deep networks ; Advani and Saxe Generalization error dynamics in high dimensions) - in the introduction, generalization is not clearly explained, which seems kind of like an important thing to do... - VC is a measure/set, not a norm? - the section "Linked to the mode selection of distribution" is unclear. I don*t see the justification for assuming that a decrease in test accuracy indicates mode selection, and don*t understand figure 4b (it*s just two bumps labelled mode 1 and mode 2 with an arrow between them. What are the axes of this plot?? Is this real data, or just a vague illustration of mode selection?) - Did you tune the learning rate for the adagrad and gradient descent methods? I find it very surprising that they would fail to converge. - experiments are not described in sufficient detail to be reproduced. What are the other hyperparameters for the various experiments? Only test set results are plotted; what about training/validation curves?