Wide adoption of complex RNN based models is hindered by their inference performance, cost and memory requirements. To address this issue, we develop AntMan, combining structured sparsity with low-rank decomposition synergistically, to reduce model computation, size and execution time of RNNs while attaining desired accuracy. AntMan extends knowledge distillation based training to learn the compressed models efficiently. Our evaluation shows that AntMan offers up to 100x computation reduction with less than 1pt accuracy drop for language and machine reading comprehension models. Our evaluation also shows that for a given accuracy target, AntMan produces 5x smaller models than the state-ofart. Lastly, we show that AntMan offers super-linear speed gains compared to theoretical speedup, demonstrating its practical value on commodity hardware.
Model Compression is used to reduce the computational and memory complexity of DL models without significantly affecting accuracy. Existing works focused on pruning and regularization based approaches where as this paper explores structured sparsity on RNNs, using predefined compact structures.
They replace matrix-vector multiplications which is the building computational block part of RNNs, with localized group projections (LGP). where LGP divides the input and output vectors into groups where the elements of the output group is computed as a linear combination of those from the corresponding input group. Moreover, they use a permutation matrix or a dense-square matrix to combine outputs across groups. They also combine LGP with low-rank matrix decomposition in order to further reduce the computations. 
Strong points: 
Paper shows how combining the SVD and LGP can reduce computation. In particular in matrix-vector multiplications Ax, low rank reduces the computation by factorizing A into smaller matrices P and Q, while LGP reduces computation by sparsifying these matrices without changing their dimensions.
The paper discussed that their model target labels alone does not generalize well on test data and they showed teacher-student training helps greatly on retaining accuracy. They use the original uncompressed model as the teacher, and train the compressed model(student) to imitate the output distribution of the teacher, in addition to training on the target labels.
Paper is well written and easy to follow. 
This paper would be much stronger if it compared against quantization and latest pruning techniques. 
This paper replace matrix-vector multiplications with the lowRank-LGP, but they only consider RNN networks. I am wondering how it affects other models given the fact that matrix-vector multiplications is the core of many deep learning models. It is not clear why their approach should only work for RNNs.
Table 1 shows the reduction in computation and model size over the original matrix-vector multiplications Ax. I think in this analysis the computation of the those approaches are neglected. For example running the SVD alone on A (n by m matrix) takes O(m^2 n+n^3). That is true that if P and Q are given, then the cost would be n(m+n)/r. However, finding P and Q takes O(m^2 n+n^3) that could be very expensive when matrices are large.
Table 2 only shows the LGB-shuffle resuts. What about the combined SVD and LGP? Similarly in Table 4, what is the performance of the LGB-Dense?