Word-embeddings are a vital component of Natural Language Processing (NLP) systems and have been extensively researched. Better representations of words have come at the cost of huge memory footprints, which has made deploying NLP models on edge-devices challenging due to memory limitations. Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment. In this paper, we propose Distilled Embedding, an (input/output) embedding compression method based on low-rank matrix decomposition with an added non-linearity. First, we initialize the weights of our decomposition by learning to reconstruct the full word-embedding and then fine-tune on the downstream task employing knowledge distillation on the factorized embedding. We conduct extensive experimentation with various compression rates on machine translation, using different data-sets with a shared word-embedding matrix for both embedding and vocabulary projection matrices. We show that the proposed technique outperforms conventional low-rank matrix factorization, and other recently proposed word-embedding matrix compression methods.
This paper proposes a method for compressing embedding matrices of both encoder/decoder embeddings.
The basic idea of the proposed method is to reconstruct the embedding matrix by what they called the “funneling decomposition” method, whose parameter shape is identical to the SVD (low-rank matrix) decomposition with additional non-linear function.
Therefore, the idea itself is not so novel and innovative.
Moreover, their method requires the embedding matrix as the teacher signal for calculating the reconstruction loss.
We need to note that the memory requirement of the proposed method during training will increase.
 
One of the notable advantages of the proposed method is that their proposed method seems to successfully reduce the embedding matrix even if it shares the parameters with the output layer, which is a de-facto standard model architecture for NMT.
As pointed out by the authors, this seems to be the first success of reducing the embedding matrix with a tied embedding setting.
1,
The authors claim that “We demonstrate that at the same compression rate our method outperforms existing state-of-the-art methods.” at the end of the Introduction section.
However, according to Tables 1, 2, and 3, it seems that the performance gain is marginal compared with similar methods.
For example, 
37.78 (proposed) <=> 37.78 (Shi & Yu (2018)) 
26.97 (proposed) <=> 26.75 (Chen et al. (2018)
and
 42.62 (proposed) <=> 42.37 (SVD with rank 64),
which are the at most 0.25 BLEU gain.
I believe that most of MT researchers hardly say that BLEU 0.25 difference is a significant improvement. Besides, the authors should perform a statistically significant test if they say “our method outperforms existing state-of-the-art methods.”
 
 
2
I am a bit confused about the following inconsistency;
The authors say that “Compressing embedding matrices without sacrificing model performance is essential for successful commercial edge deployment” in the abstract.
However, according to Table 1, the number of parameters for embeddings is 16.3M, which is only 27% of the total number of parameters in Transformer base.
By this fact, compressing embedding matrices seems not essential for successful commercial edge deployment.
 
In Table 6, it is explicitly unclear what is the difference between 
“Funneling with Emb. Distillation”, “Funneling (with non-linearity),” and “Funneling (with retraining all weights).”
Please give us a more precise explanation.