Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames. Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations. A potential solution is to extend to higher-order recurrent models. However, such a model requires a large number of parameters and operations, making it intractable to learn and prone to overfitting in practice. In this work, we propose Convolutional TensorTrain LSTM (Conv-TT-LSTM), which learns higher-order Convolutional Long Short-Term Memory (ConvLSTM) efficiently using Convolutional Tensor-Train Decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio-temporal information with low memory and computational requirements by efficient low-rank tensor-train representations. We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and other ConvLSTM-based approaches, but with much fewer parameters.
This paper proposed a convolutional tensor-train (CTT) format based high-order and convolutional LSTM approach for long-term video prediction. This paper is well-motivated. Video data usually have high dimensional input, and the proposed method aims to explicitly take into account more than one hidden representation of previous frames - both lead to a huge number of parameters. Therefore, some sort of parameter reduction is needed. This paper considers two different types of operations - convolution and tensor-train (TT) decomposition - in an interleaved way. The basic model considered in this paper is a high-order variant of convolutional LSTM (convLSTM).
 
There exist several works using tensor decomposition methods including TT to compress a fully connected layer or a convolutional layer in neural nets, to break the memory bottleneck and accelerate computation. This paper takes a different direction - it further embeds convolution into the TT decomposition and thus defines a new type of tensor decomposition, termed convolutional tensor train (CTT) decomposition. CTT is used to represent the huge weight matrices arisen in the high-order convLSTM. To my best knowledge, this combination of convolution and TT decomposition is new.
 
The paper is well-written as the literature review is well done. Experimental results demonstrate improved performance over the convolutional LSTM baseline, a fewer number of parameters, and the qualitative results show sharp and clean digits. This improvement could be attributed to multiple causes: the high-order, the tensor decomposition-based compression, or the CTT. The authors also provide an ablation study, but it mainly concerns comparisons with ConvLSTM.   
 
Despite the promising results, this paper is not ready for ICLR yet. Below is a list of suggested points needed to address:
(1) Yang et al 2017 claim that TT-RNN without convolution can also capture spatial and temporal dependence patterns in video modeling. This is an important baseline but missing in the current version of the paper. 
(2) The justification of high-order modeling in long-term prediction. The first-order model also implicitly aggregates multiple past steps. It would be good to add more experimental evidence to support the necessity of the high-order.
(3) There exists some unjustified complexity for the CTT approach. How does it compare to TT for high-order ConvLSTM?
 
Perhaps, a more complete ablation study should include:
(1) LSTM with TT but without high-order and convolution
(2) LSTM with high-order and TT but without convolution
(3) ConvLSTM with TT
(4) ConvLSTM with CTT
(5) ConvLSTM with high-order and TT
(6) ConvLSTM with high-order and CTT
 
Question:
• How is the backpropagation done for the CTT core tensors? 
• What is the error propagation issue of first-order methods and how does the high-order one not prone to it?