The intriguing role of module criticality in the generalization of deep networks
We study the phenomenon that some modules of deep neural networks (DNNs) are more emph{critical} than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network*s performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called {em module criticality}, based on the shape of the valleys that connects the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.
The paper builds upon the *module criticality* phenomenon and proposes a quantitative approach to measure this at the module and the network level. A module*s criticality is low if when it is switched to its initialization value, the error does not change drastically. The paper uses a convex combination of the initial weights and the final weights of a layer/module to define an optimization path to traverse. The authors quantitatively define the module criticality such that it depends on how much closer the weights can get to the initial weights on this path while still being robust to random permutations. The network critically is defined as the sum of the module criticality measure of all the layers. Empirical results on CIFAR10 show that the network*s criticality is reflective of the generalization performance. For example, increasing resnet depth leads to improved generalization and low criticality. Though intuitively, it is not clear why moving closer to the initial values and thus lower average criticality indicated better generalization. It will be useful ot have a discussion on this issue. Results on other datasets will also be useful. Overall, the network criticality measure appears a useful tool to predict the generalization performance compared to other measures such as distance from initialization, weight spectrum, and others.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The intriguing role of module criticality in the generalization of deep networks
We study the phenomenon that some modules of deep neural networks (DNNs) are more emph{critical} than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network*s performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called {em module criticality}, based on the shape of the valleys that connects the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper builds upon the *module criticality* phenomenon and proposes a quantitative approach to measure this at the module and the network level. A module*s criticality is low if when it is switched to its initialization value, the error does not change drastically. The paper uses a convex combination of the initial weights and the final weights of a layer/module to define an optimization path to traverse. The authors quantitatively define the module criticality such that it depends on how much closer the weights can get to the initial weights on this path while still being robust to random permutations. The network critically is defined as the sum of the module criticality measure of all the layers. Empirical results on CIFAR10 show that the network*s criticality is reflective of the generalization performance. For example, increasing resnet depth leads to improved generalization and low criticality. Though intuitively, it is not clear why moving closer to the initial values and thus lower average criticality indicated better generalization. It will be useful ot have a discussion on this issue. Results on other datasets will also be useful. Overall, the network criticality measure appears a useful tool to predict the generalization performance compared to other measures such as distance from initialization, weight spectrum, and others.
assistant
The paper builds upon the *module criticality* phenomenon and proposes a quantitative approach to measure this at the module and the network level. A module*s criticality is low if when it is switched to its initialization value, the error does not change drastically. The paper uses a convex combination of the initial weights and the final weights of a layer/module to define an optimization path to traverse. The authors quantitatively define the module criticality such that it depends on how much closer the weights can get to the initial weights on this path while still being robust to random permutations. The network critically is defined as the sum of the module criticality measure of all the layers. Empirical results on CIFAR10 show that the network*s criticality is reflective of the generalization performance. For example, increasing resnet depth leads to improved generalization and low criticality. Though intuitively, it is not clear why moving closer to the initial values and thus lower average criticality indicated better generalization. It will be useful ot have a discussion on this issue. Results on other datasets will also be useful. Overall, the network criticality measure appears a useful tool to predict the generalization performance compared to other measures such as distance from initialization, weight spectrum, and others.