Long-term Forecasting using Tensor-Train RNNs
We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions. Furthermore, we decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs. We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.
 This work addresses an important and outstanding problem: accurate long-term forecasting using deep recurrent networks. The technical approach seems well motivated, plausible, and potentially a good contribution, but the experimental work has numerous weaknesses which limit the significance of the work in current form. For one, the 3 datasets tested are not established as among the most suitable, well-recognized benchmarks for evaluating long-term forecasting. It would be far more convincing if the author’s used well-established benchmark data, for which existing best methods have already been well-tuned to get their best results. Otherwise, the reader is left with concerns that the author’s may not have used the best settings for the baseline method results reported, which indeed is a concern here (see below). One weakness with the experiments is that it is not clear that they were fair to RNN or LSTM, for example, in terms of giving them the same computation as the TT-RNNs. Section Hyper-parameter Analysis” on page 7 explains that they determined best TT rank and lags via grid search. But presumably larger values for rank and lag require more computation, so to be fair to RNN and LSTM they should be given more computation as well, for example allowing them more hidden units than TT-RNNs get, so that overall computation cost is the same for all 3 methods. As far as this reviewer can tell, the authors offer no experiments to show that a larger number of units for RNN or LSTM would not have helped them in improving long-term forecasting accuracies, so this seems like a very serious and plausible concern. Also, on page 6 the authors say that they tried ARMA but that it performed about 5% worse than LSTM, and thus dismissing direct comparisons of ARMA against TT-RNN. But they are unclear whether they gave ARMA as much hyper-parameter tuning (e.g. for number of lags) via grid search as their proposed TT-RNN benefited from. Again, the concern here is that the experiments are plausibly not being fair to all methods equally. So, due to the weaknesses in the experimental work, this work seems a bit premature. It should more clearly establish that their proposed TT-RNN are indeed performing well compared to existing SOTA.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Long-term Forecasting using Tensor-Train RNNs
We present Tensor-Train RNN (TT-RNN), a novel family of neural sequence architectures for multivariate forecasting in environments with nonlinear dynamics. Long-term forecasting in such systems is highly challenging, since there exist long-term temporal dependencies, higher-order correlations and sensitivity to error propagation. Our proposed tensor recurrent architecture addresses these issues by learning the nonlinear dynamics directly using higher order moments and high-order state transition functions. Furthermore, we decompose the higher-order structure using the tensor-train (TT) decomposition to reduce the number of parameters while preserving the model performance. We theoretically establish the approximation properties of Tensor-Train RNNs for general sequence inputs, and such guarantees are not available for usual RNNs. We also demonstrate significant long-term prediction improvements over general RNN and LSTM architectures on a range of simulated environments with nonlinear dynamics, as well on real-world climate and traffic data.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This work addresses an important and outstanding problem: accurate long-term forecasting using deep recurrent networks. The technical approach seems well motivated, plausible, and potentially a good contribution, but the experimental work has numerous weaknesses which limit the significance of the work in current form. For one, the 3 datasets tested are not established as among the most suitable, well-recognized benchmarks for evaluating long-term forecasting. It would be far more convincing if the author’s used well-established benchmark data, for which existing best methods have already been well-tuned to get their best results. Otherwise, the reader is left with concerns that the author’s may not have used the best settings for the baseline method results reported, which indeed is a concern here (see below). One weakness with the experiments is that it is not clear that they were fair to RNN or LSTM, for example, in terms of giving them the same computation as the TT-RNNs. Section Hyper-parameter Analysis” on page 7 explains that they determined best TT rank and lags via grid search. But presumably larger values for rank and lag require more computation, so to be fair to RNN and LSTM they should be given more computation as well, for example allowing them more hidden units than TT-RNNs get, so that overall computation cost is the same for all 3 methods. As far as this reviewer can tell, the authors offer no experiments to show that a larger number of units for RNN or LSTM would not have helped them in improving long-term forecasting accuracies, so this seems like a very serious and plausible concern. Also, on page 6 the authors say that they tried ARMA but that it performed about 5% worse than LSTM, and thus dismissing direct comparisons of ARMA against TT-RNN. But they are unclear whether they gave ARMA as much hyper-parameter tuning (e.g. for number of lags) via grid search as their proposed TT-RNN benefited from. Again, the concern here is that the experiments are plausibly not being fair to all methods equally. So, due to the weaknesses in the experimental work, this work seems a bit premature. It should more clearly establish that their proposed TT-RNN are indeed performing well compared to existing SOTA.
assistant
 This work addresses an important and outstanding problem: accurate long-term forecasting using deep recurrent networks. The technical approach seems well motivated, plausible, and potentially a good contribution, but the experimental work has numerous weaknesses which limit the significance of the work in current form. For one, the 3 datasets tested are not established as among the most suitable, well-recognized benchmarks for evaluating long-term forecasting. It would be far more convincing if the author’s used well-established benchmark data, for which existing best methods have already been well-tuned to get their best results. Otherwise, the reader is left with concerns that the author’s may not have used the best settings for the baseline method results reported, which indeed is a concern here (see below). One weakness with the experiments is that it is not clear that they were fair to RNN or LSTM, for example, in terms of giving them the same computation as the TT-RNNs. Section Hyper-parameter Analysis” on page 7 explains that they determined best TT rank and lags via grid search. But presumably larger values for rank and lag require more computation, so to be fair to RNN and LSTM they should be given more computation as well, for example allowing them more hidden units than TT-RNNs get, so that overall computation cost is the same for all 3 methods. As far as this reviewer can tell, the authors offer no experiments to show that a larger number of units for RNN or LSTM would not have helped them in improving long-term forecasting accuracies, so this seems like a very serious and plausible concern. Also, on page 6 the authors say that they tried ARMA but that it performed about 5% worse than LSTM, and thus dismissing direct comparisons of ARMA against TT-RNN. But they are unclear whether they gave ARMA as much hyper-parameter tuning (e.g. for number of lags) via grid search as their proposed TT-RNN benefited from. Again, the concern here is that the experiments are plausibly not being fair to all methods equally. So, due to the weaknesses in the experimental work, this work seems a bit premature. It should more clearly establish that their proposed TT-RNN are indeed performing well compared to existing SOTA.