For sequence models with large word-level vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep word-level representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves the efficiency of the Transformer model by about 1.4 times while delivering similar performance.
This paper describes an approach to learn word embedding functions more efficiently and with fewer parameters. This is done by replacing the embedding lookup function which is typical in NLP tasks such as language modeling and machine translation with a hierarchical embedding model. This allows for a low dimensional embedding layer, reducing total parameters and training time. A novel skip-connections architecture is introduced as a part of the "embedding generation model". Experiments are conducted for language modeling and machine translation tasks and performance improvements are observed with a reduction in parameters and lesser training time.
The direction of this work is nice, the problem that is being tackled is indeed important.
The obtained results are nice (though this can be improved) and there is indeed some potential value in this work.
However, I have the following concern. The paper completely ignores a lot of previous and concurrent work in reducing the size of the embedding layer. These works are in most cases not even cited and no empirical comparisons are provided. For example, please see below works in matrix factorization approaches, sparse word representation learning, codebook learning and other quantization approaches for compressing word embeddings:
https://www.aclweb.org/anthology/P16-1022/
https://aaai.org/ojs/index.php/AAAI/article/download/4578/4456
http://web.cs.ucla.edu/~chohsieh/papers/Mulcode_Compressor.pdf
https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17042/16071
https://storage.googleapis.com/pub-tools-public-publication-data/pdf/f158f7c81ed8e985fd51a20d193103ce427cad51.pdf
https://arxiv.org/pdf/1711.01068.pdf
https://arxiv.org/abs/1510.00149
I would appreciate if comparisons with some of these approaches is provided in the next iteration of this work.
Other suggestions:
1. I think the paper would benefit from some analysis of the differences in the word embeddings learnt by a general lookup table learning model in comparison with the word embeddings learnt by this model. How are the embeddings compressed? How do the decompressed embeddings compare to the embeddings learnt by the lookup approach? More insights in the machinery via some visualizations would help.
2. How do the gains of this method change as more or less training data is provided. For example, are the gains lesser on Gigaword? This would be interesting to know.
3. GLT is mentioned twice in this paper. Perhaps a slightly more detailed explanation of the same would help improve the readability of this paper.
Based on the presentation improvements and new experiments added to the paper in the rebuttal time period, I am updating my evaluation of this work.