A SINGLE SHOT PCA-DRIVEN ANALYSIS OF NETWORK STRUCTURE TO REMOVE REDUNDANCY
Deep learning models have outperformed traditional methods in many fields such as natural language processing and computer vision. However, despite their tremendous success, the methods of designing optimal Convolutional Neural Networks (CNNs) are still based on heuristics or grid search. The resulting networks obtained using these techniques are often overparametrized with huge computational and memory requirements. This paper focuses on a structured, explainable approach towards optimal model design that maximizes accuracy while keeping computational costs tractable. We propose a single-shot analysis of a trained CNN that uses Principal Component Analysis (PCA) to determine the number of filters that are doing significant transformations per layer, without the need for retraining. It can be interpreted as identifying the dimensionality of the hypothesis space under consideration. The proposed technique also helps estimate an optimal number of layers by looking at the expansion of dimensions as the model gets deeper. This analysis can be used to design an optimal structure of a given network on a dataset, or help to adapt a predesigned network on a new dataset. We demonstrate these techniques by optimizing VGG and AlexNet networks on CIFAR-10, CIFAR-100 and ImageNet datasets.
 This paper proposes a set of heuristics for identifying a good neural network architecture, based on PCA of unit activations over the dataset. The method shows modest accuracy drops and improved parameter counts in several settings. Major comments: The proposed method is not optimal, in the sense that it does not provide any guarantees on the loss in performance or generalization error. It is largely heuristic, particularly the notion that the depth can be determined to be the layer at which expansion of the representation stops. This itself depends on the initial architecture one uses. There is no clear theoretical link between PCA of the filters and performance of the network. In general, important information in features could come from low variance directions. Moreover, the PCA step is meant to give the dimensionality of the transformations, but this will depend on the choice of nonlinearity. Optimizing the method’s design parameter (% variance retained) may be comparably difficult as optimizing the number of filters in the networks directly. The experiments show that the proposed method can often reduce the number of operations and parameters, at usually slight decreases in test performance. The experiments could be greatly improved by adding comparison to related parameter reduction mechanisms. It is hard to evaluate the benefit of the proposed method without these. Additionally, the drop in performance highlights the question of what exactly is being optimized if these design principles are ‘optimal.’ It could be interesting to look at the patterns in reduction of filters in the experimental results run here. Do adjacent layers typically increase the number of filters by certain factors after pruning? The paper is mostly clear but the high level organization could be improved. Overall the proposed method offers a simple heuristic for choosing the architecture of a deep network, but this seems unlikely to yield principled improvements at present.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A SINGLE SHOT PCA-DRIVEN ANALYSIS OF NETWORK STRUCTURE TO REMOVE REDUNDANCY
Deep learning models have outperformed traditional methods in many fields such as natural language processing and computer vision. However, despite their tremendous success, the methods of designing optimal Convolutional Neural Networks (CNNs) are still based on heuristics or grid search. The resulting networks obtained using these techniques are often overparametrized with huge computational and memory requirements. This paper focuses on a structured, explainable approach towards optimal model design that maximizes accuracy while keeping computational costs tractable. We propose a single-shot analysis of a trained CNN that uses Principal Component Analysis (PCA) to determine the number of filters that are doing significant transformations per layer, without the need for retraining. It can be interpreted as identifying the dimensionality of the hypothesis space under consideration. The proposed technique also helps estimate an optimal number of layers by looking at the expansion of dimensions as the model gets deeper. This analysis can be used to design an optimal structure of a given network on a dataset, or help to adapt a predesigned network on a new dataset. We demonstrate these techniques by optimizing VGG and AlexNet networks on CIFAR-10, CIFAR-100 and ImageNet datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes a set of heuristics for identifying a good neural network architecture, based on PCA of unit activations over the dataset. The method shows modest accuracy drops and improved parameter counts in several settings. Major comments: The proposed method is not optimal, in the sense that it does not provide any guarantees on the loss in performance or generalization error. It is largely heuristic, particularly the notion that the depth can be determined to be the layer at which expansion of the representation stops. This itself depends on the initial architecture one uses. There is no clear theoretical link between PCA of the filters and performance of the network. In general, important information in features could come from low variance directions. Moreover, the PCA step is meant to give the dimensionality of the transformations, but this will depend on the choice of nonlinearity. Optimizing the method’s design parameter (% variance retained) may be comparably difficult as optimizing the number of filters in the networks directly. The experiments show that the proposed method can often reduce the number of operations and parameters, at usually slight decreases in test performance. The experiments could be greatly improved by adding comparison to related parameter reduction mechanisms. It is hard to evaluate the benefit of the proposed method without these. Additionally, the drop in performance highlights the question of what exactly is being optimized if these design principles are ‘optimal.’ It could be interesting to look at the patterns in reduction of filters in the experimental results run here. Do adjacent layers typically increase the number of filters by certain factors after pruning? The paper is mostly clear but the high level organization could be improved. Overall the proposed method offers a simple heuristic for choosing the architecture of a deep network, but this seems unlikely to yield principled improvements at present.
assistant
 This paper proposes a set of heuristics for identifying a good neural network architecture, based on PCA of unit activations over the dataset. The method shows modest accuracy drops and improved parameter counts in several settings. Major comments: The proposed method is not optimal, in the sense that it does not provide any guarantees on the loss in performance or generalization error. It is largely heuristic, particularly the notion that the depth can be determined to be the layer at which expansion of the representation stops. This itself depends on the initial architecture one uses. There is no clear theoretical link between PCA of the filters and performance of the network. In general, important information in features could come from low variance directions. Moreover, the PCA step is meant to give the dimensionality of the transformations, but this will depend on the choice of nonlinearity. Optimizing the method’s design parameter (% variance retained) may be comparably difficult as optimizing the number of filters in the networks directly. The experiments show that the proposed method can often reduce the number of operations and parameters, at usually slight decreases in test performance. The experiments could be greatly improved by adding comparison to related parameter reduction mechanisms. It is hard to evaluate the benefit of the proposed method without these. Additionally, the drop in performance highlights the question of what exactly is being optimized if these design principles are ‘optimal.’ It could be interesting to look at the patterns in reduction of filters in the experimental results run here. Do adjacent layers typically increase the number of filters by certain factors after pruning? The paper is mostly clear but the high level organization could be improved. Overall the proposed method offers a simple heuristic for choosing the architecture of a deep network, but this seems unlikely to yield principled improvements at present.