Finding Deep Local Optima Using Network Pruning
Artificial neural networks (ANNs) are very popular nowadays and offer reliable solutions to many classification problems. However, training deep neural networks (DNN) is time-consuming due to the large number of parameters. Recent research indicates that these DNNs might be over-parameterized and different solutions have been proposed to reduce the complexity both in the number of parameters and in the training time of the neural networks. Furthermore, some researchers argue that after reducing the neural network complexity via connection pruning, the remaining weights are irrelevant and retraining the sub-network would obtain a comparable accuracy with the original one. This may hold true in most vision problems where we always enjoy a large number of training samples and research indicates that most local optima of the convolutional neural networks may be equivalent. However, in non-vision sparse datasets, especially with many irrelevant features where a standard neural network would overfit, this might not be the case and there might be many non-equivalent local optima. This paper presents empirical evidence for these statements and an empirical study of the learnability of neural networks (NNs) on some challenging non-linear real and simulated data with irrelevant variables. Our simulation experiments indicate that the cross-entropy loss function on XOR-like data has many local optima, and the number of local optima grows exponentially with the number of irrelevant variables. We also introduce a connection pruning method to improve the capability of NNs to find a deep local minimum even when there are irrelevant variables. Furthermore, the performance of the discovered sparse sub-network degrades considerably either by retraining from scratch or the corresponding original initialization, due to the existence of many bad optima around. Finally, we will show that the performance of neural networks for real-world experiments on sparse datasets can be recovered or even improved by discovering a good sub-network architecture via connection pruning.
The paper addresses the very important topic of local minima in Deep Learning. This is one of the central questions in the theory of Deep Learning for the last years, and despite many interesting results the main questions remain wide open. The reviewer really likes the approach proposed in the paper, to use a simple model and an artificially generated data to study a certain phenomenon. The reviewer represents the opinion that more focus on such setups would greatly benefit the community in terms of progressing the theoretical understanding. The claim made in the paper that there is a relationship between the number/suboptimality of local minima and the scarcity of the data is both convincing and interesting. The result is well motivated and explained. What the reviewer thinks the paper would greatly benefit from would be improving the Related Work section. There was a lot of valuable work in the field done in the past years that ids very relevant to the results presented that is not mentioned.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Finding Deep Local Optima Using Network Pruning
Artificial neural networks (ANNs) are very popular nowadays and offer reliable solutions to many classification problems. However, training deep neural networks (DNN) is time-consuming due to the large number of parameters. Recent research indicates that these DNNs might be over-parameterized and different solutions have been proposed to reduce the complexity both in the number of parameters and in the training time of the neural networks. Furthermore, some researchers argue that after reducing the neural network complexity via connection pruning, the remaining weights are irrelevant and retraining the sub-network would obtain a comparable accuracy with the original one. This may hold true in most vision problems where we always enjoy a large number of training samples and research indicates that most local optima of the convolutional neural networks may be equivalent. However, in non-vision sparse datasets, especially with many irrelevant features where a standard neural network would overfit, this might not be the case and there might be many non-equivalent local optima. This paper presents empirical evidence for these statements and an empirical study of the learnability of neural networks (NNs) on some challenging non-linear real and simulated data with irrelevant variables. Our simulation experiments indicate that the cross-entropy loss function on XOR-like data has many local optima, and the number of local optima grows exponentially with the number of irrelevant variables. We also introduce a connection pruning method to improve the capability of NNs to find a deep local minimum even when there are irrelevant variables. Furthermore, the performance of the discovered sparse sub-network degrades considerably either by retraining from scratch or the corresponding original initialization, due to the existence of many bad optima around. Finally, we will show that the performance of neural networks for real-world experiments on sparse datasets can be recovered or even improved by discovering a good sub-network architecture via connection pruning.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper addresses the very important topic of local minima in Deep Learning. This is one of the central questions in the theory of Deep Learning for the last years, and despite many interesting results the main questions remain wide open. The reviewer really likes the approach proposed in the paper, to use a simple model and an artificially generated data to study a certain phenomenon. The reviewer represents the opinion that more focus on such setups would greatly benefit the community in terms of progressing the theoretical understanding. The claim made in the paper that there is a relationship between the number/suboptimality of local minima and the scarcity of the data is both convincing and interesting. The result is well motivated and explained. What the reviewer thinks the paper would greatly benefit from would be improving the Related Work section. There was a lot of valuable work in the field done in the past years that ids very relevant to the results presented that is not mentioned.
assistant
The paper addresses the very important topic of local minima in Deep Learning. This is one of the central questions in the theory of Deep Learning for the last years, and despite many interesting results the main questions remain wide open. The reviewer really likes the approach proposed in the paper, to use a simple model and an artificially generated data to study a certain phenomenon. The reviewer represents the opinion that more focus on such setups would greatly benefit the community in terms of progressing the theoretical understanding. The claim made in the paper that there is a relationship between the number/suboptimality of local minima and the scarcity of the data is both convincing and interesting. The result is well motivated and explained. What the reviewer thinks the paper would greatly benefit from would be improving the Related Work section. There was a lot of valuable work in the field done in the past years that ids very relevant to the results presented that is not mentioned.