GENERALIZED TENSOR MODELS FOR RECURRENT NEURAL NETWORKS
Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency -a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN. Such networks, however, are not very often applied to real life tasks. In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency. Our theoretical results are verified by a series of extensive computational experiments.
The authors extend the theoretical results of a paper previously presented in the last edition of ICLR (2018), where it was demonstrated that Recurrent Neural Network can be interpreted as a tensor network decomposition based on the Tensor-Train (TT, Oseledets et al, 2011).
While previous results covered the multiplicative nonlinearity only, the contribution of the current paper is the extension of the analysis of universality and depth efficiency (Cohen et al, 2016) to different nonlinearities, for example ReLU (Rectified Linear Unit), which is very important from the practical point of view.
The paper is well written and have a good structure. However, I found that some deep concepts are not well introduced, and maybe other more trivial results are discussed with unnecessary details. The following comments could help authors to improve the quality of presentation of their paper:
-	Section 3.1 (Score Functions and Feature Tensor) is a bit short and difficult to read. 
o	Maybe, a more motivating introduction could be included in order to justify the definition of score functions (eq. 2). 
o	It would be also nice to state that, according to eq. (3), the feature tensor is a rank-1 tensor. 
o	I would suggest moving the definition of outer product to the Appendix, since most readers know it very well.
o	It is said that eq. 2 possesses the universal approximation property (it can approximate any function with any prescribed precision given sufficiently large M). It is not clear which is the approximation function.
-	A Connection with Tensor-Ring (TR) format, if possible, could be helpful: It is known that TR format (Zhao et al, 2016, arXiv:1606.05535), which is obtained by connecting the first and last units in a TT model, helps to alleviate the requirement of large ranks in the first and last the core tensors of a TT model reaching to a decomposition with an evenly distributed rank bounds. I think, it would be interesting to make a connection of RNN to TR because the assumption of R_i < R for all i becomes more natural. I would like to see at least some comment from the authors about the applicability of TR in the context of analysis of RNN, if possible. Maybe, also the initial hidden state defined in page 5 can be avoided if TR is used instead of TT.
-	Fig 2 shows that Test accuracy of a shallow network (CP based) is lower and increases with the number of parameters approaching to the one for RNN (TT based). It would be necessary to show the results for an extended range in the number of parameters, for example, by plotting the results up to 10^6. It is expected that, at some point, the effect of overfitting start decreasing the test accuracy.
-       When scores functions are presented (eq. 2) it is written the term "logits" between brackets. Could you please clarify why this term is introduced here? Usually, logit of a probability p is defined as L(p)=p/(1-p). What is the usage of this term in this work? 
-      I think the theory is presented for a model with the two-classes only but used for multiple classes in the experimental sections. It should be necessary to make some comment about this in the paper.
-      Details about how the RNN based on TT is applied must be added. More specifically, the authors should provide answers to clarify the following questions: 
(i) Are patches overlapped or non-overlapped? 
(ii) What value of M is used? and is there any general rule for this choice? 
(iii) How the classification in the 10-classes is obtained? Are you using a softmax function in the last layer? Are you using one weight tensor W_c per class (c=1,2,...,10). Please provide these technical details. 
(iv) Please, specify which nonlinear activation sigma is used in the feature map f_\theta(x).
(v) How many feature maps are used? and, Are the matrix A and vector b learned from training dataset or only the TT-cores need to be learned?