Learnability of Learned Neural Networks
This paper explores the simplicity of learned neural networks under various settings: learned on real vs random data, varying size/architecture and using large minibatch size vs small minibatch size. The notion of simplicity used here is that of learnability i.e., how accurately can the prediction function of a neural network be learned from labeled samples from it. While learnability is different from (in fact often higher than) test accuracy, the results herein suggest that there is a strong correlation between small generalization errors and high learnability. This work also shows that there exist significant qualitative differences in shallow networks as compared to popular deep networks. More broadly, this paper extends in a new direction, previous work on understanding the properties of learned neural networks. Our hope is that such an empirical study of understanding learned neural networks might shed light on the right assumptions that can be made for a theoretical study of deep learning.
 Review Summary: The primary claim that there is "a strong correlation between small generalization errors and high learnability" is correct and supported by evidence, but it doesn*t provide much insight for the questions posed at the beginning of the paper or for a general better understanding of theoretical deep learning. In fact the relationship between test accuracy and learnability seems quite obvious, which unfortunately undermines the usefulness of the learnability metric which is used in many experiments in the paper. For example, consider the results in Table 7. A small network (N1 = 16 neurons) with low test accuracy results in a low learnability, while a large network (N1 = 1024 neurons) gets a higher test accuracy and higher learnability. In this case, the small network can be thought of as applying higher label noise relative to the larger network. Thus it is expected that agreement between N1 and N2 (learnability) will be higher for the larger network, as the predictions of N1 are less noisy. More importantly, this relationship between test accuracy and learnability doesn*t answer the original question Q2 posed: "Do larger neural networks learn simpler patterns compared to neural networks when trained on real data". It instead draws some obvious conclusions about noisy labeling of training data. Other results presented in the paper are puzzling and require further experimentation and discussion, such as the trend that the learnability of shallow networks on random data is much higher than 10%, as discussed at the bottom of page 4. The authors provide some possible reasoning, stating that this strange effect could be due to class imbalance, but it isn*t convincing enough. Other comments: -Section 3.4 is unrelated to the primary arguments of the paper and seems like a filler. -Equations should have equation numbers -Learnability numbers reported in all tables should be between 0-1 per the definition on page 3 -As suggested in the final sentence of the discussion, it would be nice if conclusions drawn from the learnability experiments done in this paper were applied to the design new networks which better generalize
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learnability of Learned Neural Networks
This paper explores the simplicity of learned neural networks under various settings: learned on real vs random data, varying size/architecture and using large minibatch size vs small minibatch size. The notion of simplicity used here is that of learnability i.e., how accurately can the prediction function of a neural network be learned from labeled samples from it. While learnability is different from (in fact often higher than) test accuracy, the results herein suggest that there is a strong correlation between small generalization errors and high learnability. This work also shows that there exist significant qualitative differences in shallow networks as compared to popular deep networks. More broadly, this paper extends in a new direction, previous work on understanding the properties of learned neural networks. Our hope is that such an empirical study of understanding learned neural networks might shed light on the right assumptions that can be made for a theoretical study of deep learning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Review Summary: The primary claim that there is "a strong correlation between small generalization errors and high learnability" is correct and supported by evidence, but it doesn*t provide much insight for the questions posed at the beginning of the paper or for a general better understanding of theoretical deep learning. In fact the relationship between test accuracy and learnability seems quite obvious, which unfortunately undermines the usefulness of the learnability metric which is used in many experiments in the paper. For example, consider the results in Table 7. A small network (N1 = 16 neurons) with low test accuracy results in a low learnability, while a large network (N1 = 1024 neurons) gets a higher test accuracy and higher learnability. In this case, the small network can be thought of as applying higher label noise relative to the larger network. Thus it is expected that agreement between N1 and N2 (learnability) will be higher for the larger network, as the predictions of N1 are less noisy. More importantly, this relationship between test accuracy and learnability doesn*t answer the original question Q2 posed: "Do larger neural networks learn simpler patterns compared to neural networks when trained on real data". It instead draws some obvious conclusions about noisy labeling of training data. Other results presented in the paper are puzzling and require further experimentation and discussion, such as the trend that the learnability of shallow networks on random data is much higher than 10%, as discussed at the bottom of page 4. The authors provide some possible reasoning, stating that this strange effect could be due to class imbalance, but it isn*t convincing enough. Other comments: -Section 3.4 is unrelated to the primary arguments of the paper and seems like a filler. -Equations should have equation numbers -Learnability numbers reported in all tables should be between 0-1 per the definition on page 3 -As suggested in the final sentence of the discussion, it would be nice if conclusions drawn from the learnability experiments done in this paper were applied to the design new networks which better generalize
assistant
 Review Summary: The primary claim that there is "a strong correlation between small generalization errors and high learnability" is correct and supported by evidence, but it doesn*t provide much insight for the questions posed at the beginning of the paper or for a general better understanding of theoretical deep learning. In fact the relationship between test accuracy and learnability seems quite obvious, which unfortunately undermines the usefulness of the learnability metric which is used in many experiments in the paper. For example, consider the results in Table 7. A small network (N1 = 16 neurons) with low test accuracy results in a low learnability, while a large network (N1 = 1024 neurons) gets a higher test accuracy and higher learnability. In this case, the small network can be thought of as applying higher label noise relative to the larger network. Thus it is expected that agreement between N1 and N2 (learnability) will be higher for the larger network, as the predictions of N1 are less noisy. More importantly, this relationship between test accuracy and learnability doesn*t answer the original question Q2 posed: "Do larger neural networks learn simpler patterns compared to neural networks when trained on real data". It instead draws some obvious conclusions about noisy labeling of training data. Other results presented in the paper are puzzling and require further experimentation and discussion, such as the trend that the learnability of shallow networks on random data is much higher than 10%, as discussed at the bottom of page 4. The authors provide some possible reasoning, stating that this strange effect could be due to class imbalance, but it isn*t convincing enough. Other comments: -Section 3.4 is unrelated to the primary arguments of the paper and seems like a filler. -Equations should have equation numbers -Learnability numbers reported in all tables should be between 0-1 per the definition on page 3 -As suggested in the final sentence of the discussion, it would be nice if conclusions drawn from the learnability experiments done in this paper were applied to the design new networks which better generalize