We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions. Furthermore, we decompose the higherorder structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs. We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.
The paper proposes Tensor-Train RNN and Tensor-Train LSTM (TT-RNN/TLSTM), a RNN/LSTM architecture whose hidden unit at time t h_t is computed from the tensor-vector product between a tensor of weights and a concatenation of hidden units from the previous L time steps. The motivation is to incorporate previous hidden states and high-order correlations among them to better predict long-term temporal dependencies for seq2seq problems. To address the issue of the number of parameters growing exponentially in the rank of the tensor, the model uses a low rank decomposition called the ‘tensor-train decomposition’ to make the number of parameters linear in the rank. Some theoretical analysis on the number of hidden units required for a given estimation error, and experimental results have been provided for synthetic and real sequential data.
First of all, the presentation of the method in section 2.1 is confusing and there seem to be various ambiguities in the notation that harms understanding of the method. The tensor-vector product in equation (6) appears problematic. The notation that I think is standard is as follows: given a tensor W \in R^{n_1 \times … \times n_P} and vectors v_p \in R^{n_p}, the tensor-vector product W \times_{p=1}^P v_p = vec(W) \otimes_{p=1}^P v_p = \sum{i_1,...,i_P} \prod_{p=1}^P v_{p,i_p}. So I’m guessing you want to get rid of the \otimes signs (the kronecker products) in (6) or you want to remove the summation and write W \times_{p=1}^P s_{t-1}. Also \alpha that appears in (6) is never defined. Is it another index? This is confusing because you say W is P-dimensional but have P+1 indices for it including alpha (W_{\alpha i_1 … i_p}). Moreover the dimensionality of W^{hx} x_t in (6) is R^H judging from the notation in page 2, but isn’t the tensor-vector product a scalar? Also am I correct in thinking that s_{t-1} should be [1, h_{t-1}^T, …, h_{t-L}^T], i.e. a vector of length LH+1 rather than a matrix? The notation from page 2 implies that you are using column vectors, so the definition of s_{t-1} makes it appear as an (L+1) by H matrix, which could make the reader interpret s_{t-1;i_1} in (6) as vectors instead of scalars (this is reinforced by the kronecker product between these s_{t-1;i_p}). I had to work this out from the number of parameters (HL+1)^P in section 2.2. The diagram of s_{t-1} in Figure 3 is also confusing, because it isn’t obvious that the unlabelled grey bars are copies of s_{t-1}. Also I notice that the name ‘Tensor Train RNN/LSTM’ has been used in Yang et al, 2017. You probably want to avoid using the same name since the models are different. It would be nice if you could explain in a bit more detail about how they are different in the related work section.
Assuming I have understood the method correctly, the idea of using tensor products to incorporate higher order interactions between the hidden states at different times appears sensible. From the theoretical analysis, you claim that 1) smoother f is easier to approximate, and 2) polynomial interactions are more efficient than linear ones. The first point seems fairly self-explanatory and doesn’t seem to require a proof. The second point isn’t so convincing because you have two additive terms on the right hand side of the inequality in Theorem 3.1 (btw I’m guessing you want the inequality to be the other way round): the first term is independent of p, and the second decreases exponentially with p. Your second point would only hold if this first term is reasonably small, but this doesn’t seem obvious to me.
Regarding the experiments, I’m sceptical as to whether a grid search over hyperparameters for TLSTM vs grid search over the same hyperparameters for (M)LSTM provides a fair comparison. You probably want to compare the models given the same number of parameters, since given the same state size, TLSTM will have many more parameters than (M)LSTM. A plot of x-axis: # parameters, y-axis: average RMSE at convergence would be informative. Moreover for figure 8, you probably want to control the time taken for training instead of just comparing validation loss at the same number of steps. I imagine the best performing TLSTM model will have many more parameters and hence take much longer to train than the best performing LSTM model. 
Moreover, it seems as though the increased prediction accuracy from LSTM is marginal considering you have 3 more hyperparameters to tune (L,S,P - what was the value of P used for the experiments?) and that tuning them is important to prevent overfitting.
I’m also curious as to how TLSTM compares to hierarchical RNN approaches for modelling long-term dependencies. It will be interesting to compare against models like Stacked LSTM (Graves, 2013), Grid LSTM (Kalchbrenner, 2015) and HM LSTM (Chung, 2017). These models have mostly been evaluated on text, but I don’t see any reason they can’t be extended to sequential forecasting on time series data. Also regularisation techniques such as batch-norm for LSTMs (Cooijmans et al, 2016) and layer-norm (Ba et al, 2016) seem to help a lot for increasing prediction accuracy. Did you investigate these techniques to control overfitting?
Other minor comments on presentation:
For figure 6, the legends are inconsistent with the caption. Also you might want to overlay predictions on top of the ground truth for better comparison and also to save space.
Overall, I think there are vast scopes for improvement in presentation and comparisons with other methods, and hence find the paper not yet ready for publication.