LEARNING TO SHARE: SIMULTANEOUS PARAMETER TYING AND SPARSIFICATION IN DEEP LEARNING
Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, making both storage and computation very expensive. This has motivated a large body of work to reduce the complexity of the neural network by using sparsity-inducing regularizers. Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weights are forced to share a common value. Some forms of weight sharing are hard-wired to express certain in- variances, with a notable example being the shift-invariance of convolutional layers. However, there may be other groups of weights that may be tied together during the learning process, thus further re- ducing the complexity of the network. In this paper, we adopt a recently proposed sparsity-inducing regularizer, named GrOWL (group ordered weighted l1), which encourages sparsity and, simulta- neously, learns which groups of parameters should share a common value. GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates. Unlike standard sparsity-inducing regularizers (e.g., l1 a.k.a. Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identifies strongly correlated neurons by tying the corresponding weights to a common value. This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the training process to simultaneously identify significant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e., keeping only the significant neurons and enforcing the learned tying structure. We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance.
 The authors propose to use the group ordered weighted l1 regulariser (GrOWL) combined with clustering of correlated features to select and tie parameters, leading to a sparser representation with a reduced parameter space. They apply the proposed method two well-known benchmark datasets under a fully connected and a convolutional neural network, and demonstrate that in the former case a slight improvement in accuracy can be achieved, while in the latter, the method performs similar to the group-lasso, but at a reduced computational cost for classifying new images due to increased compression of the network. The paper is well written and motivated, and the idea seems fairly original, although the regularisation approach itself is not new. Like many new approaches in this field, it is hard to judge from this paper and its two applications alone whether the approach will lead to significant benefits in general, but it certainly seems promising. Positive points: - Demonstrated improved compression with similar performance to the standard weighted decay method. - Introduced a regularization technique that had not been previously used in this field, and that improves on the group lasso in terms of compression, without apparent loss of accuracy. - Applied an efficient proximal gradient algorithm to train the model. Negative points: - The method is sold as inducing a clustering, but actually, the clustering is a separate step, and the choice of clustering algorithm might well have an influence on the results. It would have been good to see more discussion or exploration of this. I would not claim that, for example, the fused lasso is a clustering algorithm for regression coefficients, even though it demonstrably sets some coefficients to the same value, so it seems wrong to imply the same for GrOWL. - In the example applications, it is not clear how the accuracy was obtained (held-out test set? cross-validation?), and it would have been good to get an estimate for the variance of this quantity, to see if the differences between methods are actually meaningful (I suspect not). Also, why is the first example reporting accuracy, but the second example reports error? - There is a slight contradiction in the abstract, in that the method is introduced as guarding against overfitting, but then the last line states that there is "slight or even no loss on generalization performance". Surely, if we reduce overfitting, then by definition there would have to be an improvement in generalization performance, so should we draw the conclusion that the method has not actually been demonstrated to reduce overfitting? Minor point: - p.5, in the definition of prox_Q(epsilon), the subscript for the argmin should be nu, not theta.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
LEARNING TO SHARE: SIMULTANEOUS PARAMETER TYING AND SPARSIFICATION IN DEEP LEARNING
Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, making both storage and computation very expensive. This has motivated a large body of work to reduce the complexity of the neural network by using sparsity-inducing regularizers. Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weights are forced to share a common value. Some forms of weight sharing are hard-wired to express certain in- variances, with a notable example being the shift-invariance of convolutional layers. However, there may be other groups of weights that may be tied together during the learning process, thus further re- ducing the complexity of the network. In this paper, we adopt a recently proposed sparsity-inducing regularizer, named GrOWL (group ordered weighted l1), which encourages sparsity and, simulta- neously, learns which groups of parameters should share a common value. GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates. Unlike standard sparsity-inducing regularizers (e.g., l1 a.k.a. Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identifies strongly correlated neurons by tying the corresponding weights to a common value. This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the training process to simultaneously identify significant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e., keeping only the significant neurons and enforcing the learned tying structure. We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors propose to use the group ordered weighted l1 regulariser (GrOWL) combined with clustering of correlated features to select and tie parameters, leading to a sparser representation with a reduced parameter space. They apply the proposed method two well-known benchmark datasets under a fully connected and a convolutional neural network, and demonstrate that in the former case a slight improvement in accuracy can be achieved, while in the latter, the method performs similar to the group-lasso, but at a reduced computational cost for classifying new images due to increased compression of the network. The paper is well written and motivated, and the idea seems fairly original, although the regularisation approach itself is not new. Like many new approaches in this field, it is hard to judge from this paper and its two applications alone whether the approach will lead to significant benefits in general, but it certainly seems promising. Positive points: - Demonstrated improved compression with similar performance to the standard weighted decay method. - Introduced a regularization technique that had not been previously used in this field, and that improves on the group lasso in terms of compression, without apparent loss of accuracy. - Applied an efficient proximal gradient algorithm to train the model. Negative points: - The method is sold as inducing a clustering, but actually, the clustering is a separate step, and the choice of clustering algorithm might well have an influence on the results. It would have been good to see more discussion or exploration of this. I would not claim that, for example, the fused lasso is a clustering algorithm for regression coefficients, even though it demonstrably sets some coefficients to the same value, so it seems wrong to imply the same for GrOWL. - In the example applications, it is not clear how the accuracy was obtained (held-out test set? cross-validation?), and it would have been good to get an estimate for the variance of this quantity, to see if the differences between methods are actually meaningful (I suspect not). Also, why is the first example reporting accuracy, but the second example reports error? - There is a slight contradiction in the abstract, in that the method is introduced as guarding against overfitting, but then the last line states that there is "slight or even no loss on generalization performance". Surely, if we reduce overfitting, then by definition there would have to be an improvement in generalization performance, so should we draw the conclusion that the method has not actually been demonstrated to reduce overfitting? Minor point: - p.5, in the definition of prox_Q(epsilon), the subscript for the argmin should be nu, not theta.
assistant
 The authors propose to use the group ordered weighted l1 regulariser (GrOWL) combined with clustering of correlated features to select and tie parameters, leading to a sparser representation with a reduced parameter space. They apply the proposed method two well-known benchmark datasets under a fully connected and a convolutional neural network, and demonstrate that in the former case a slight improvement in accuracy can be achieved, while in the latter, the method performs similar to the group-lasso, but at a reduced computational cost for classifying new images due to increased compression of the network. The paper is well written and motivated, and the idea seems fairly original, although the regularisation approach itself is not new. Like many new approaches in this field, it is hard to judge from this paper and its two applications alone whether the approach will lead to significant benefits in general, but it certainly seems promising. Positive points: - Demonstrated improved compression with similar performance to the standard weighted decay method. - Introduced a regularization technique that had not been previously used in this field, and that improves on the group lasso in terms of compression, without apparent loss of accuracy. - Applied an efficient proximal gradient algorithm to train the model. Negative points: - The method is sold as inducing a clustering, but actually, the clustering is a separate step, and the choice of clustering algorithm might well have an influence on the results. It would have been good to see more discussion or exploration of this. I would not claim that, for example, the fused lasso is a clustering algorithm for regression coefficients, even though it demonstrably sets some coefficients to the same value, so it seems wrong to imply the same for GrOWL. - In the example applications, it is not clear how the accuracy was obtained (held-out test set? cross-validation? ), and it would have been good to get an estimate for the variance of this quantity, to see if the differences between methods are actually meaningful (I suspect not). Also, why is the first example reporting accuracy, but the second example reports error? - There is a slight contradiction in the abstract, in that the method is introduced as guarding against overfitting, but then the last line states that there is "slight or even no loss on generalization performance". Surely, if we reduce overfitting, then by definition there would have to be an improvement in generalization performance, so should we draw the conclusion that the method has not actually been demonstrated to reduce overfitting? Minor point: - p.5, in the definition of prox_Q(epsilon), the subscript for the argmin should be nu, not theta.