Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames.Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations. A potential solution is to extend to higher-order spatio-temporal recurrent models. However, such a model requires a large number of parameters and operations, making it intractable to learn in practice and is prone to overfitting. In this work, we propose convolutional tensor-train LSTM (Conv-TT-LSTM), which learns higher-orderConvolutional LSTM (ConvLSTM) efficiently using convolutional tensor-train decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio-temporal information at a small cost of memory and computation by using efficient low-rank tensor representations. We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and better/comparable results to other ConvLSTM-based approaches, but with much fewer parameters.
This paper build a higher-order spatio-temporal model by means of combining Convolutional Tensor-Train Decomposition(CTTD) and ConvLSTM, and utilize the combination method to solve long-term video prediction problems. The CTTD factorizes a large convolutional kernel into a chain of smaller tensors, so as to relieve the difficult convergence and overfitting problems caused by too much model params. Experiments on Moving-MNIST and KTH datasets show that the proposed method achieved better results than standard ConvLSTM, and in some way comparable with SOTA model. Ablation Studies are also provided. Although it seems novel combing CTTD with ConvLSTM, the idea of CTTD and the combination mainly comes from [Yu et al.,2017] and [Yang et al.,2017]?this paper use the method in a new problem of video prediction, I think the theoretical innovation is not enough for ICLR. Although the experimental results were better than ConvLSTM(2015), but not as good as PredRNN++(2018), especially in terms of the MSE metrics. Since the prediction accuracy has not yet achieved, I don*t think the reduction of model params is a matter of primary importance. What’s more, Moving-MNIST and KTH are relatively simple datasets, video prediction on a more complicated datasets such as UCF101 will be more convincing. Conclusion: This paper is in some way novel, but not enough for ICLR, and the experiment results seems not enough convincing, so I will give a weak reject.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames.Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations. A potential solution is to extend to higher-order spatio-temporal recurrent models. However, such a model requires a large number of parameters and operations, making it intractable to learn in practice and is prone to overfitting. In this work, we propose convolutional tensor-train LSTM (Conv-TT-LSTM), which learns higher-orderConvolutional LSTM (ConvLSTM) efficiently using convolutional tensor-train decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio-temporal information at a small cost of memory and computation by using efficient low-rank tensor representations. We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and better/comparable results to other ConvLSTM-based approaches, but with much fewer parameters.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper build a higher-order spatio-temporal model by means of combining Convolutional Tensor-Train Decomposition(CTTD) and ConvLSTM, and utilize the combination method to solve long-term video prediction problems. The CTTD factorizes a large convolutional kernel into a chain of smaller tensors, so as to relieve the difficult convergence and overfitting problems caused by too much model params. Experiments on Moving-MNIST and KTH datasets show that the proposed method achieved better results than standard ConvLSTM, and in some way comparable with SOTA model. Ablation Studies are also provided. Although it seems novel combing CTTD with ConvLSTM, the idea of CTTD and the combination mainly comes from [Yu et al.,2017] and [Yang et al.,2017]?this paper use the method in a new problem of video prediction, I think the theoretical innovation is not enough for ICLR. Although the experimental results were better than ConvLSTM(2015), but not as good as PredRNN++(2018), especially in terms of the MSE metrics. Since the prediction accuracy has not yet achieved, I don*t think the reduction of model params is a matter of primary importance. What’s more, Moving-MNIST and KTH are relatively simple datasets, video prediction on a more complicated datasets such as UCF101 will be more convincing. Conclusion: This paper is in some way novel, but not enough for ICLR, and the experiment results seems not enough convincing, so I will give a weak reject.
assistant
This paper build a higher-order spatio-temporal model by means of combining Convolutional Tensor-Train Decomposition(CTTD) and ConvLSTM, and utilize the combination method to solve long-term video prediction problems. The CTTD factorizes a large convolutional kernel into a chain of smaller tensors, so as to relieve the difficult convergence and overfitting problems caused by too much model params. Experiments on Moving-MNIST and KTH datasets show that the proposed method achieved better results than standard ConvLSTM, and in some way comparable with SOTA model. Ablation Studies are also provided. Although it seems novel combing CTTD with ConvLSTM, the idea of CTTD and the combination mainly comes from [Yu et al.,2017] and [Yang et al.,2017]?this paper use the method in a new problem of video prediction, I think the theoretical innovation is not enough for ICLR. Although the experimental results were better than ConvLSTM(2015), but not as good as PredRNN++(2018), especially in terms of the MSE metrics. Since the prediction accuracy has not yet achieved, I don*t think the reduction of model params is a matter of primary importance. What’s more, Moving-MNIST and KTH are relatively simple datasets, video prediction on a more complicated datasets such as UCF101 will be more convincing. Conclusion: This paper is in some way novel, but not enough for ICLR, and the experiment results seems not enough convincing, so I will give a weak reject.