Published as a conference paper at ICLR 2018 LEARNING TO SHARE: SIMULTANEOUS PARAMETER TYING AND SPARSIFICATION IN DEEP LEARNING
Deep neural networks (DNNs) may contain millions, even billions, of parameters/weights, making storage and computation very expensive and motivating a large body of work aimed at reducing their complexity by using, e.g., sparsityinducing regularization. Parameter sharing/tying is another well-known approach for controlling the complexity of DNNs by forcing certain sets of weights to share a common value. Some forms of weight sharing are hard-wired to express certain invariances; a notable example is the shift-invariance of convolutional layers. However, other groups of weights may be tied together during the learning process to further reduce the network complexity. In this paper, we adopt a recently proposed regularizer, GrOWL (group ordered weighted 1 ), which encourages sparsity and, simultaneously, learns which groups of parameters should share a common value. GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates. Unlike standard sparsity-inducing regularizers (e.g., 1 a.k.a. Lasso), GrOWL not only eliminates unimportant neurons by setting all their weights to zero, but also explicitly identifies strongly correlated neurons by tying the corresponding weights to a common value. This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization during training to simultaneously identify significant neurons and groups of parameters that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e., keeping only the significant neurons and enforcing the learned tying structure. We evaluate this approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization accuracy.
SUMMARY
The paper proposes to apply GrOWL regularization to the tensors of parameters between each pair of layers. The groups are composed of all coefficients associated to inputs coming from the same neuron in the previous layer. The proposed algorithm is a simple proximal gradient algorithm using the proximal operator of the GrOWL norm. Given that the GrOWL norm tend to empirically reinforce a natural clustering of the vectors of coefficients which occurs in some layers, the paper proposes to cluster the corresponding parameter vectors, to replace them with their centroid and to retrain with the constrain that some vectors are now equal. Experiments show that some sparsity is obtained by the model and that together with the clustering and high compression of the model is obtained which maintaining or improving over a good level of generalization accuracy. In comparison, plain group Lasso yields compressed versions that are too sparse, and tend to degrade performance. The method is also competitive with weight decay with much better compression.
REVIEW
Given the well known issue that the Lasso tends to select arbitrarily and in a non stable way variables
that are correlated *but* given that the well known elastic-net (and conceptually simpler than GrOWL) was proposed to address that issue already more than 10 years ago, it would seem relevant to compare the proposed method with the group elastic-net.
The proposed algorithm is a simple proximal gradient algorithm, but since the objective is non-convex it would be relevant to provide references for convergence guarantees of the algorithm.
How should the step size eta be chosen? I don't see that this is discussed in the paper.
In the clustering algorithm how is the threshold value chosen?
Is it chosen by cross validation?
Is the performance better with clustering or without?
Is the same threshold chosen for GrOWL and the Lasso?
It would be useful to know which values of p, Lambda_1 and Lambda_2 are selected in the experiments?
For Figures 5,7,8,9 given that the matrices do not have particular structures that need to be visualized but that the important thing to compare is the distribution of correlation between pairs, these figures that are hard to read and compare would be advantageously replaced by histograms of the values of the correlations between pairs (of different variables). Indeed, right now one must rely on comparison of shades of colors in the thin lines that display correlation and it is really difficult to appreciate how much of correlation of what level are present in each Figure. Histograms would extract exactly the relevant information...
A brief description of affinity propagation, if only in the appendix, would be relevant.
Why this method as opposed to more classical agglomerative clustering?
A brief reminder of what the principle of weight decay is would also be relevant for the paper to more self contained.
The proposed experiments are compelling, except for the fact that it would be nice to have a comparison with the group elastic-net. 
I liked figure 6.d and would vote for inclusion in the main paper.
TYPOS etc 
3rd last line of sec. 3.2 can fail at selecting -> fail to select
In eq. (5) theta^t should be theta^{(t)}
In section 4.1 you that the network has a single fully connected layer of hidden units -> what you mean is that the network has a single hidden layer, which is furthermore fully connected.
You cite several times Sergey (2015) in section 4.2. It seems you have exchanged first name and last name plus the corresponding reference is quite strange.
Appendix B line 5 ", while." -> incomplete sentence.