Published as a conference paper at ICLR 2018 INITIALIZATION MATTERS: ORTHOGONAL PREDICTIVE STATE RECURRENT NEURAL NETWORKS
Learning to predict complex time-series data is a fundamental challenge in a range of disciplines including Machine Learning, Robotics, and Natural Language Processing. Predictive State Recurrent Neural Networks (PSRNNs) (Downey et al.,  2017)  are a state-of-the-art approach for modeling time-series data which combine the benefits of probabilistic filters and Recurrent Neural Networks into a single model. PSRNNs leverage the concept of Hilbert Space Embeddings of distributions (Smola et al., 2007) to embed predictive states into a Reproducing Kernel Hilbert Space, then estimate, predict, and update these embedded states using Kernel Bayes Rule. Practical implementations of PSRNNs are made possible by the machinery of Random Features, where input features are mapped into a new space where dot products approximate the kernel well. Unfortunately PSRNNs often require a large number of RFs to obtain good results, resulting in large models which are slow to execute and slow to train. Orthogonal Random Features (ORFs) (Yu et al., 2016) is an improvement on RFs which has been shown to decrease the number of RFs required for pointwise kernel approximation. Unfortunately, it is not clear that ORFs can be applied to PSRNNs, as PSRNNs rely on Kernel Ridge Regression as a core component of their learning algorithm, and the theoretical guarantees of ORF do not apply in this setting. In this paper, we extend the theory of ORFs to Kernel Ridge Regression and show that ORFs can be used to obtain Orthogonal PSRNNs (OPSRNNs), which are smaller and faster than PSRNNs. In particular, we show that OPSRNN models clearly outperform LSTMs and furthermore, can achieve accuracy similar to PSRNNs with an order of magnitude smaller number of features needed.
I was very confused by some parts of the paper that are simple copy-past from the paper of Downey et al.  which has been accepted for publication in NIPS. In particular, in section 3, several sentences are taken as they are from the Downey et al.’s paper. Some examples :
« provide a compact representation of a dynamical system
by representing state as a set of predictions of features of future observations. » 
« a predictive state is defined as… , where…  is a vector of features of future observations and ...  is a vector of
features of historical observations. The features are selected such that ...  determines the distribution
of future observations … Filtering is the process of mapping a predictive state… »
Even the footnote has been copied & pasted: « For convenience we assume that the system is k-observable: that is, the distribution of all future observations
is determined by the distribution of the next k observations. (Note: not by the next k observations
themselves.) At the cost of additional notation, this restriction could easily be lifted. »
«  This approach is fast, statistically consistent, and reduces to simple
linear algebra operations. » 
Normally, I should have stopped reviewing, but I decided to continue  since those parts only concerned the preliminaries part.
A key element in PSRNN is to used as an initialization a kernel ridge regression. The main result here, is to show that using orthogonal random features approximates well the original kernel comparing to random fourrier features as considered in PSRNN. This result is formally stated and proved in the paper.
The paper comes with some experiments in order to empirically demonstrate the superiority  orthogonal random features over RFF. Three data sets are considered (Swimmer,  Mocap and  Handwriting). 
I found it that the contribution of the paper is very limited. The connexion to PSRNN is very tenuous since the main results are about the regression part. in Theorems 2 and 3 there are no mention to PSRNN.
Also the experiment is not very convincing. The datasets are too small with observations in low dimensions, and I found it not very fair to consider LSTM in such settings.
Some minor remarks:
- p3: We use RFs-> RFFs
- p5: ||X||, you mean |X| the size of the dataset
- p12: Eq (9). You need to add « with probability $1-\rho$ as in Avron’s paper.
- p12: the derivation of Eq (10) from Eq (9) needs to be detailed.   
I thank the author for their detailed answers. Some points have been clarified but other still raise issues. In particular, I continue thinking that the contribution is limited. Accordingly, I did not change my scores.