Our work addresses two important issues with recurrent neural networks: (1) they are over-parametrized, and (2) the recurrent weight matrix is ill-conditioned. The former increases the sample complexity of learning and the training time. The latter causes the vanishing and exploding gradient problem. We present a flexible recurrent neural network model called Kronecker Recurrent Units (KRU). KRU achieves parameter efficiency in RNNs through a Kronecker factored recurrent matrix. It overcomes the ill-conditioning of the recurrent matrix by enforcing soft unitary constraints on the factors. Thanks to the small dimensionality of the factors, maintaining these constraints is computationally efficient. Our experimental results on seven standard data-sets reveal that KRU can reduce the number of parameters by three orders of magnitude in the recurrent weight matrix compared to the existing recurrent models, without trading the statistical performance. These results in particular show that while there are advantages in having a high dimensional recurrent space, the capacity of the recurrent part of the model can be dramatically reduced.
Summary of the paper
-------------------------------
This paper proposes to factorize the hidden-to-hidden matrix of RNNs into a Kronecker product of small matrices, thus reducing the number of parameters, without reducing the size of the hidden vector. They also propose to use a soft unitary constraint on those small matrices (which is equivalent to a soft unitary constraint on the Kronecker product of those matrices), that is fast to compute. They evaluate their model on 6 small scale RNN experiments.
Clarity, Significance and Correctness
--------------------------------------------------
Clarity: The main idea is clearly motivated and presented, but the experiment section failed to convince me (see details below).
Significance: The idea of using factorization for RNNs is not particularly novel. However, it is really nice to be able to decouple the hidden size and the number of recurrent parameters in a simple way. Also, the combination of Kronecker product and soft unitary constraint is really interesting.
Correctness: There are minor flaws. Some of the baselines seems to perform poorly, and some comparisons with the baselines seems unfair (see the questions below).
Questions
--------------
1. Section 3: You say that you can vary 'pf' and 'qf' to set the trade-off between computational budget and performances. Have you run some experiments where you vary those parameters?
2. Section 4: Are you using the soft unitary constraint in your experiments? Do you have an hyper-parameter that sets the amplitude of the constraint? If yes, what is its value? Are you using it also on the vanilla RNN or the LSTM?
3. Section 4.1: You say that you don't train the recurrent matrix in the KRU version. Do you also not train the recurrent matrix in the other models (RNN, LSTM,...)? If yes, how do you explain the differences? If no, I don't see how those curves compare.
4. Section 4.3: Why does your LSTM in pMNIST performs so poorly? There are way better curves reported in the literature (eg in "Unitary Evolution Recurrent Neural Netwkrs" or "Recurrent Batch Normalization").
5. General: How does your method compares with other factorization approaches, such as in "Factorization Tricks for LSTM Networks"?
6. Section 4: How does the KRU compares to the other parametrizations, in term of wall-clock time?
Remarks
------------
The main claim of the paper is that RNN are over-parametrized and take a long time to train (which I both agree with), but you didn't convinced me that your parametrization solve any of those problems. I would suggest to:
1. Compare more clearly setups where you fix the hidden size.
2. Compare more clearly setups where you fix the number of parameters.
With systematic comparisons like that, it would be easier to understand where the gains in performances are coming from.
3. Add an experiment where you vary 'pf' and 'qf' (and keep the hidden size fixed) to show how the optimization/generalization performances can be tweaked.
4. Add computation time (wall-clock) for all the experiments, to see how it compares in practice (this could definitively weight in your favor, since you seems to have a nice CUDA implementation).
5. Present results on larger-scale applications (Text8, Teaching Machines to Read and Comprehend, 3 layers LSTM speech recognition setup on TIMIT, DRAW, Machine Translation, ...), especially because your method is really easy to plug in any existing code available online.
Typos / Form
------------------
1. sct 1, par 3: "using Householder reflection vectors, it allows a fine-grained" -> "using Householder reflection vectors, which allows a fine-grained"
2. sct 1, par 3: "This work called as Efficient" -> "This work, called Efficient"
5. sct 1, par 5: "At the heart of KRU is the use of Kronecker" -> "At the heart of KRU, we use Kronecker"
6. sct 1, par 5: "Thanks to the properties of Kronecker matrices" -> "Thanks to the properties of the Kronecker product"
7. sct 1, par 5: "vanilla real space RNN" -> "vanilla RNN"
8. sct 2, par 1: "Consider a standard recurrent" -> "Consider a standard vanilla recurrent"
9. sct 2, par 1: "step t RNN" -> "step t, a vanilla RNN"
11. sct 2.1, par 1: "U and V, this is efficient using modern BLAS" -> "U and V, which can be efficiently computed using modern BLAS"
12. sct 2.3, par 2: "matrices have a determinant of 1 or −1, i.e., the set of all rotations and reflections respectively" -> "matrices, i.e., the set of all rotations and reflections, have a determinant of 1 or −1."
13. sct 3, par 1: "are called as Kronecker" -> "are called Kronecker"
14. sct 3, par 3: "used it's spectral" -> "used their spectral"
15. sct 3, par 3: "Kronecker matrices" -> "Kronecker products"
18. sct 4.4, par 3: "parameters are increased" -> "parameters increases"
19. sct 5: There is some more typos in the conclusion ("it's" -> "its")
20. Some plots are hard to read / interpret, mostly because of the round "ticks" you use on the curves. I suggest you remove them everywhere. Also, in the adding problem, it would be cleaner if you down-sampled a bit the curves (as they are super noisy). In pixel by pixel MNIST, some of the legends might have some typos (FC uRNN), and you should use "N" instead of "n" to be consistent with the notation of the paper.
21. Appendix A to E are not necessary, since they are from the literature.
22. sct 3.1, par 2: "is approximately unitary." -> "is approximately unitary (cf Appendix F)."
23. sct 4, par 1: "and backward operations." -> "and backward operations (cf Appendix G and H)."
Pros
------
1. Nice Idea that allows to decouple the hidden size with the number of hidden-to-hidden parameters.
2. Cheap soft unitary constraint
3. Efficient CUDA implementation (not experimentally verified)
Cons
-------
1. Some experimental setups are unfair, and some other could be clearer
2. Only small scale experiments (although this factorization has huge potential on larger scale experiments)
3. No wall-clock time that show the speed of the proposed parametrization.