The embedding layers transforming input words into real vectors are the key components of deep neural networks used in natural language processing. However, when the vocabulary is large, the corresponding weight matrices can be enormous, which precludes their deployment in a limited resource setting. We introduce a novel way of parametrizing embedding layers based on the Tensor Train (TT) decomposition, which allows compressing the model significantly at the cost of a negligible drop or even a slight gain in performance. We evaluate our method on a wide range of benchmarks in natural language processing and analyze the trade-off between performance and compression ratios for a wide range of architectures, from MLPs to LSTMs and Transformers.
This paper proposes a low-rank tensor decomposition model (Tensor Train-TT [Oseledets et al, 2011]) to parameterize the embedding matrix in Natural Language Processing (NLP). It shows that TT allows for a compression of the network and sometimes even a slight increase of test accuracy. The paper is well written and easy to follow.
I found the idea as a natural consequence of many recent papers proposing tensor decomposition to parameterize deep learning networks. However, I think this is the first time that the concept has been applied to learning an embedding matrix, which is an important problem in the field.
The authors reported several experimental results on different tasks and datasets for NLP such as: Sentiment Analysis, Neural Machine Translation and Language Modeling; and an application to the click through rate prediction problem.
I think the paper is of limited novelty but includes interesting experimental results that helps to better understand the potential and limitations of tensor decompositions in deep learning architectures.
Below, I summarize the issues I found, and I would like the authors to address them in their responses:
Major issues:
-	In Page 4 (and Appendix B), the authors show a comparison with Tensor Ring (TR) and conclude that TT marginally outperforms TR in terms of BLEU measure for a fixed number of parameters in both models. I found this comparison incomplete, weak and misleading because of the following reasons:
o	TR is a more general model including TT as a particular case when the first and last ranks are equal to one (Zhao et al, 2016). In fact, in this experiment, the authors chose all intermediate ranks set at the same value R with the first/last ranks set to 1 and R for TT and TR, respectively, which is not a fair comparison. Shown results suggest that first/last ranks contain less information than intermediate ranks, but it would be necessary to explore other combinations of rank values without keeping them constant to explore the generalization power of the TR model including TT as a particular case.
o	The authors compares TT and TR only in case of the NMT, Transformer-big on WMT‘14 English-to-German dataset, where the results are not good for TT and TR. It is noted that baseline model (Big) attains a Sacre BLEU = 28.84, TT1 = 28.53 and TR1 = 28.07 and the compression rate is only 210/179=1.17 for TT1 and TR1 and the Iteration time is larger than in the baseline model. In this case, there is no a clear advantage of using TT or TR.
In my opinion, to improve the paper, I think the authors could:
o	To avoid the sentence “In our experiments, however, the resulting TR embeddings performed slightly worse than TT–embeddings with the same number of parameters” unless more conclusive and exhaustive experiments are performed comparing TT and TR.
o	To add some comparison results between TT and TR for the rest of datasets such as Sentiment Analysis, Language Modeling and the click through rate prediction problem. 
o	Highlight that TT is a particular case of TR so considering the first and last ranks equal to one reduce the number of parameters but can affect the generalization power of the model.
-	The approach of the paper is mostly intuitive. A theoretical result about why the low rank TT is able to catch the useful information of an optimal or suboptimal embedding matrix is missing.
Minor issues:
-	In last paragraph of section 3.2: The number of parameters is computed on the 3D tensor cores only. I think the size of the first/last 2D cores should be added. Please revise the equation.
-	The pseudocode for the mapping one index to multiple indices is trivial and could be avoided. If it is kept, I think the reverse operation should be also included, i.e. how to map multiple indices i1, …., iN to one index i.
-	The discussion and Figure 2 about the Gaussianity of the values in the higher order tensor based on Gaussian core tensors is not relevant. Maybe, the authors should better motivate why it is important to highlight that the distribution tends to a Gaussian density for increasing ranks.
-	In page 5, it is mentioned that “factors should be as close to each other as possible” but there is no a justification for it. Could you give some theoretical insight on why it is important to obtain uniform distribution of matrix size?
-	Section 4.1, reference to the Stanford sentiment treebank (SST) is missing.
On Nov 16th: I am satisfied with the responses provided by the Authors who made few changes to solve some identified minor issues. Thanks for taking the review report into account. I have raised the rating.