Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simpler and novel update scheme to maintain orthogonal recurrent weight matrices without using complex valued matrices. This is done by parametrizing with a skew-symmetric matrix using the Cayley transform. Such a parametrization is unable to represent matrices with negative one eigenvalues, but this limitation is overcome by scaling the recurrent weight matrix by a diagonal matrix consisting of ones and negative ones. The proposed training scheme involves a straightforward gradient calculation and update step. In several experiments, the proposed scaled Cayley orthogonal recurrent neural network (scoRNN) achieves superior results with fewer trainable parameters than other unitary RNNs.
This manuscript introduce a scheme for learning the recurrent parameter matrix in a neural network that uses the Cayley transform and a scaling weight matrix. This scheme leads to good performance on sequential data tasks and requires fewer parameters than other techniques
Comments:
-- It’s not clear to me how D is determined for each test. Given the definition in Theorem 3.1 it seems like you would have to have some knowledge of how many eigenvalues in W you expect to be close to -1. 
-- For the copying and adding problem test cases, it might be useful to clarify or cite something clarifying that the failure mode RNNs run into with temporal ordering problems is an exploding gradient, rather than any other pathological training condition, just to make it clear why these experiments are relevant.
-- The ylabel in Figure 1 is “Test Loss” which I didn’t see defined. Is this test loss the cross entropy? If so, I think it would be more effective to label the plot with that.
-- The plots in figure 1 and 2 have different colors to represent the same set of techniques. I would suggest keeping a  consistent color scheme
-- It looks like in Figure 1 the scoRNN is outperformed by the uRNN in the long run in spite of the scoRNN convergence being smoother, which should be clarified.
-- It looks like in Figure 2 the scoRNN is outperformed by the LSTM across the board, which should be clarified.
-- How is test set accuracy defined in section 5.3? Classifying digits? Recreating digits? 
-- When discussing table 1, the manuscript mentions scoRNN and Restricted-capacity uRNN have similar performance for 16k parameters and then state that scoRNN has the best test accuracy at 96.2%. However, there is no example for restricted-capacity uRNN with 69k parameters to show that the performance of restricted-capacity uRNN doesn't also increase similarly with more parameters.
-- Overall it’s unclear to me how to completely determine the benefit of this technique over the others because, for each of the tests, different techniques may have superior performance. For instance, LSTM performs best in 5.2 and in 5.3 for the MNIST test accuracy. scoRNN and Restricted-capacity uRNN perform similarly for permuted MNIST Test Accuracy in 5.3. Finally, scoRNN seems to far outperform the other techniques in table 2 on the TIMIT speech dataset. I don’t understand the significance of each test and why the relative performance of the techniques vary from one to the other.
-- For example, the manuscript seems to be making the case that the scoRNN gradients are more stable than those of a uRNN, but all of the results are presented in terms of network accuracy and not gradient stability. You can sort of see that generally the convergence is more gradual for the scoRNN than the uRNN from the training graphs but it'd be nice if there was an actual comparison of the stability of the gradients during training (as in Figure 4 of the Arjovsky 2016 paper being compared to for instance) just to make it really clear.