Convolution neural networks typically consist of many convolutional layers followed by several fully-connected layers. While convolutional layers map between high-order activation tensors, the fully-connected layers operate on flattened activation vectors. Despite its success, this approach has notable drawbacks. Flattening discards the multi-dimensional structure of the activations, and the fullyconnected layers require a large number of parameters. We present two new techniques to address these problems. First, we introduce tensor contraction layers which can replace the ordinary fully-connected layers in a neural network. Second, we introduce tensor regression layers, which express the output of a neural network as a low-rank multi-linear mapping from a high-order activation tensor to the softmax layer. Both the contraction and regression weights are learned end-to-end by backpropagation. By imposing low rank on both, we use significantly fewer parameters. Experiments on the ImageNet dataset show that applied to the popular VGG and ResNet architectures, our methods significantly reduce the number of parameters in the fully connected layers (about 65% space savings) while negligibly impacting accuracy.
This paper incorporates tensor decomposition and tensor regression into CNN by replacing its flattening operations and fully-connected layers with a new tensor regression layer. 
Pros:
The low-rank representation of tensors is able to reduce the model complexity in the original CNN without sacrificing much prediction accuracy. This is promising as it enables the implementation of complex deep learning algorithms on mobile devices due to its huge space saving performance.  Overall, this paper is easy to follow. 
Cons: 
Q1: Can the authors discuss the computational time of the proposed tensor regression layers and compare it to that of the baseline CNN? The tensor regression layer is computationally more expensive than the flattening operations in original CNN. Usually, it also involves expensive model selection procedure to choose the tuning parameters (N+1 ranks and a L2 norm sparsity parameter). In the experiments, the authors simply tried a few ranks without serious tuning. 
Q2: The authors reported the space saving in Table 1 but not in Table 2. Since spacing saving is a major contribution of the proposed method, can authors add the space saving percentage in Table 2?
Q3: There are a few typos in the current paper. I would suggest the authors to take a careful proofreading. For example,
(1) In the “Related work“ paragraph on page 2, “Lebedev et al. (2014) proposes…” should be “Lebedev et al. (2014) propose…”. Many other references have the same issue. 
(2) In Figure 1, the letter $X$ should be $\tilde{\cal X}$.
(3) In expression (5) on page 3, the core tensor is denoted by $\tilde{\cal G}$. Is this the same as $\tilde{\cal X}^{‘}$ in Figure 1?
(4) In expression (5) on page 3, the core tensor $\tilde{\cal G}$ is of dimension $(D_0, R_1, \ldots, R_N)$. However, in expression (8) on page 5, $\tilde{\cal G}$ is of dimension $(R_0, R_1, \ldots, R_N, R_{N+1})$.
(5) Use \cite{} and \citep{} correctly. For example, in the “Related work“ paragraph on page 2,
“Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models Guo et al. (2012); Rabusseau & Kadri (2016); Zhou et al. (2013); Yu & Liu (2016).”
should be
“Several prior papers address the power of tensor regression to preserve natural multi-modal structure and learn compact predictive models (Guo et al., 2012; Rabusseau & Kadri, 2016; Zhou et al., 2013; Yu & Liu, 2016).”