Workshop track -ICLR 2018 LEARNING EFFICIENT TENSOR REPRESENTATIONS WITH RING STRUCTURE NETWORKS
Tensor train (TT) decomposition is a powerful representation for high-order tensors, which has been successfully applied to various machine learning tasks in recent years. In this paper, we propose a more generalized tensor decomposition with ring structure network by employing circular multilinear products over a sequence of lower-order core tensors, which is termed as TR representation. Several learning algorithms including blockwise ALS with adaptive tensor ranks and SGD with high scalability are presented. Furthermore, the mathematical properties are investigated, which enables us to perform basic algebra operations in a computationally efficiently way by using TR representations. Experimental results on synthetic signals and real-world datasets demonstrate the effectiveness of TR model and the learning algorithms. In particular, we show that the structure information and high-order correlations within a 2D image can be captured efficiently by employing an appropriate tensorization and TR decomposition.
The paper addresses the problem of tensor decomposition which is relevant and interesting. The paper proposes Tensor Ring (TR) decomposition which improves over and bases on the Tensor Train (TT) decomposition method. TT decomposes a tensor in to a sequences of latent tensors where the first and last tensors are a 2D matrices. 
The proposed TR method generalizes TT in that the first and last tensors are also 3rd-order tensors instead of 2nd-order. I think such generalization is interesting but the innovation seems to be very limited. 
The paper develops three different kinds of solvers for TR decomposition, i.e., SVD, ALS and SGD. All of these are well known methods. 
Finally, the paper provides experimental results on synthetic data (3 oscillated functions) and image data (few sampled images). I think the paper could be greatly improved by providing more experiments and ablations to validate the benefits of the proposed methods.
Please refer to below for more comments and questions.
-- The rating has been updated.
Pros:
1. The topic is interesting.
2. The generalization over TT makes sense.
Cons:
1. The writing of the paper could be improved and more clear: the conclusions on inner product and F-norm can be integrated into "Theorem 5". And those "theorems" in section 4 are just some properties from previous definitions; they are not theorems. 
2. The property of TR decomposition is that the tensors can be shifted (circular invariance). This is an interesting property and it seems to be the major strength of TR over TT. I think the paper could be significantly improved by providing more applications of this property in both theory and experiments.
3. As the number of latent tensors increase, the ALS method becomes much worse approximation of the original optimization. Any insights or results on the optimization performance vs. the number of latent tensors?
4. Also, the paper mentions Eq. 5 (ALS) is optimized by solving d subproblems alternatively. I think this only contains a single round of optimization. Should ALS be applied repeated (each round solves d problems) until convergence?
5. What is the memory consumption for different solvers?
6. SGD also needs to update at least d times for all d latent tensors. Why is the complexity O(r^3) independent of the parameter d?
7. The ALS is so slow (if looking at the results in section 5.1), which becomes not practical. The experimental part could be improved by providing more results and description about a guidance on how to choose from different solvers.
8. What does "iteration" mean in experimental results such as table 2? Different algorithms have different cost for "each iteration" so comparing that seems not fair. The results could make more sense by providing total time consumptions and time cost per iteration. also applies to table 4.
9. Why is the \epsion in table 3 not consistent? Why not choose \epsion = 9e-4 and \epsilon=2e-15 for tensorization?
10. Also, table 3 could be greatly improved by providing more ablations such as results for (n=16, d=8), (n=4, d=4), etc. That could help readers to better understand the effect of TR.
11. Section 5.3 could be improved by providing a curve (compression vs. error) instead of just providing a table of sampled operating points.
12. The paper mentions the application of image representation but only experiment on 32x32 images. How does the proposed method handle large images? Otherwise, it does not seem to be a practical application.
13. Figure 5: Are the RSE measures computed over the whole CIFAR-10 dataset or the displayed images?
Minor:
- Typo: Page 4 Line 7 "Note that this algorithm use the similar strategy": use -> uses