Recent advances in the sparse neural network literature have made it possible to prune many large feed forward and convolutional networks with only a small quantity of data. Yet, these same techniques often falter when applied to the problem of recovering sparse recurrent networks. These failures are quantitative: when pruned with recent techniques, RNNs typically obtain worse performance than they do under a simple random pruning scheme. The failures are also qualitative: the distribution of active weights in a pruned LSTM or GRU network tend to be concentrated in specific neurons and gates, and not well dispersed across the entire architecture. We seek to rectify both the quantitative and qualitative issues with recurrent network pruning by introducing a new recurrent pruning objective derived from the spectrum of the recurrent Jacobian. Our objective is data efficient (requiring only 64 data points to prune the network), easy to implement, and produces 95 % sparse GRUs that significantly improve on existing baselines. We evaluate on sequential MNIST, Billion Words, and Wikitext.
Notes: 
  -RNN network pruning has proven to be challenging using the techniques often used with other network types.  
  -One issue is that the performance of an LSTM/GRU can hinge on a few activated gates and can lead to more concentration of influence than would be seen in a feedforward network without parameter sharing between layers.  
  -New objective uses 64 points to prune the network (I assume this is just the size of the minibatch).  
  -Result is a 95% sparse GRU cell.  
  -New idea is based on keeping weights that propagate information through many time steps.  
  -Encourages "singular values of the temporal jacobian with respect to network weights to be non-degenerate" (I suppose this means that the gradient flowing through time will contain multiple directions of variation)  
  -Introduction does a good job of introducing the key ideas.  
  -J_t is a temporal jacobian of size N x N (N is number units) at time step t.  
  -Chi is the spectral norm of this temporal jacobian.  I'm a bit confused by this, because my understanding is that the spectral norm is the largest singular value, but equation 3 looks like a sum over singular values, making it more like a frobenius norm?  
  -This jacobian isn't tractable, so paper approximates it using a first-order taylor expansion.  So basically the pruning just amounts to taking parameters with the largest gradient?  
  -Section 2.4 is confusing and seems to come out of nowhere.  Is this suggesting that the technique isn't just pruning but adding a new normalization scheme?  On second reading, this is a normalization scheme effecting which parameters to prune.  The motivation for why the gradients are normalized like this is still confusing.  If you're willing to make a linear assumption, it seems like it's enough to consider the gradient on the parameter multiplied by the magnitude of the parameter to see the overall effect of removing it?  
  -The results look good, but sequential mnist is a bit of a toy task.  I'd also like to see a more fine-grained analysis showing the tradeoff between the number of units removed and the performance.  
  -The paper claims that L2 pruning requires more data, but it's unclear if this really matters since the whole dataset was used to train both methods initially.  
  -On Table 2 the results of the technique don't seem that much better than "Random".  
  Review: This paper presented a fast pruning algorithm for RNNs, which uses the norm of the gradient as a guide to pruning.  I'm borderline on this paper.  The idea of using the gradient is good, but the explanation of some aspects like the normalization is confusing and felt random.  Additionally the results, while better than some other pruning techniques on RNNs, don't seem to be that much better than random.