SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks
We propose a fast approximation method of a softmax function with a very large vocabulary using singular value decomposition (SVD). SVD-softmax targets fast and accurate probability estimation of the topmost probable words during inference of neural network language models. The proposed method transforms the weight matrix used in the calculation of the output vector by using SVD. The approximate probability of each word can be estimated with only a small part of the weight matrix by using a few large singular values and the corresponding elements for most of the words. We applied the technique to language modeling and neural machine translation and present a guideline for good approximation. The algorithm requires only approximately 20% of arithmetic operations for an 800K vocabulary case and shows more than a three-fold speedup on a GPU.
This paper proposes an efficient way to approximate the softmax computation for large vocabulary applications. The idea is to decompose the output matrix with singular value decomposition. Then, by selecting the most important singular values you select the most probable words and also compute the partition function for a limited amount of words. These are supposed to contribute for the most part of the sum. For the remaining words, their contributions to the partition function is only approximated. 
The idea is nice and the experimental results show a significant inference speed up, along with similar performance. 
My main concern is that sometimes the paper is poorly written. Just one example: the authors use the term "logit" to denote the exponential of a dot product. This is not correct. The logit function is the inverse of the sigmoidal "logistic" function. This choice of term is therefore incorrect and misleading. 
More examples are given below. However the idea is both simple and nice. 
- Page 2, line 73: Moreover, self normalized models requires a huge training time.
- Page 3, line 93: It is not "approximately". It's the same. The softmax function outputs the probability distribution over the whole vocabulary. 
- Line 102: reword this sentence 
- Line 114: for obtaining accurate results -> to obtain accurate results
- Exactly instead of correctly in line 117
Page 5: Why you didn't use the full softmax on the wiki data ? The vocabulary is small and the computational cost is really cheap!
Page 6: The script to compute the BLEU score is included in the Moses released, but it's not the Moses toolkit. However, which one did you use ?
To have a stronger baseline you could use bpe to decompose words. Moreover, it could be more meaningful to have a larger vocabulary. For instance if you translate from English to German, the output vocabulary is really larger, even if you use bpe. I understand the goal is to show that your svd softmax performs as well as the full softmax with a huge gain in time. However, this gain of time could benefit to the NMT system by allowing a longer training in a state of the art setup with the svd softmax.