Efficient Softmax Approximation for GPUs
We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computational complexity. Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax.
The authors introduce an adaptive softmax approximation tailored for faster performance on GPUs. The key idea, which is very sensible, is to use a class-based hierarchical softmax, but where the clusters/hierarchy are distributed such that the resulting matrix multiplications are optimally-sized for GPU computation, based on their empirical tests. Their results indicate that the system does indeed work very well. In terms of presentation, I found the paper to have both clear and unclear elements. Fortunately, the underlying concepts and logic seem quite clear. Unfortunately, at various points, the writing is not. There are various minor typos (as mentioned by AnonReviewer2, in addition to some other spots, e.g. the notation describing recurrent network in Section 3 mentions an x_t which is surely different from the x_t used in the previous paragraph on regular feedforward NN*s, i think it belonged in the equation for h_t; the use of the two matrices A and P in Eq2 is strange, etc). Also, while Section 4.2 (Intuition for 2-cluster case) was a good idea to include and helpful, and while the *concepts* underlying the complexity analysis were straightforward, it could be made a lot clearer by (a) adding an additional figure such as Figure 2, along with (b) a few well-placed additional sentences unpacking the logic of the argument into easier-to-follow steps. For example, it was only when I saw Eq (6) and (7) combined with Fig(2) that the analysis on the previous page made more sense in terms of arriving at the eq for the complexity of putting the head of the distribution in the root of the tree. (Perhaps an Appendix might be the most appropriate place to add such an explanation).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Efficient Softmax Approximation for GPUs
We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computational complexity. Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors introduce an adaptive softmax approximation tailored for faster performance on GPUs. The key idea, which is very sensible, is to use a class-based hierarchical softmax, but where the clusters/hierarchy are distributed such that the resulting matrix multiplications are optimally-sized for GPU computation, based on their empirical tests. Their results indicate that the system does indeed work very well. In terms of presentation, I found the paper to have both clear and unclear elements. Fortunately, the underlying concepts and logic seem quite clear. Unfortunately, at various points, the writing is not. There are various minor typos (as mentioned by AnonReviewer2, in addition to some other spots, e.g. the notation describing recurrent network in Section 3 mentions an x_t which is surely different from the x_t used in the previous paragraph on regular feedforward NN*s, i think it belonged in the equation for h_t; the use of the two matrices A and P in Eq2 is strange, etc). Also, while Section 4.2 (Intuition for 2-cluster case) was a good idea to include and helpful, and while the *concepts* underlying the complexity analysis were straightforward, it could be made a lot clearer by (a) adding an additional figure such as Figure 2, along with (b) a few well-placed additional sentences unpacking the logic of the argument into easier-to-follow steps. For example, it was only when I saw Eq (6) and (7) combined with Fig(2) that the analysis on the previous page made more sense in terms of arriving at the eq for the complexity of putting the head of the distribution in the root of the tree. (Perhaps an Appendix might be the most appropriate place to add such an explanation).
assistant
The authors introduce an adaptive softmax approximation tailored for faster performance on GPUs. The key idea, which is very sensible, is to use a class-based hierarchical softmax, but where the clusters/hierarchy are distributed such that the resulting matrix multiplications are optimally-sized for GPU computation, based on their empirical tests. Their results indicate that the system does indeed work very well. In terms of presentation, I found the paper to have both clear and unclear elements. Fortunately, the underlying concepts and logic seem quite clear. Unfortunately, at various points, the writing is not. There are various minor typos (as mentioned by AnonReviewer2, in addition to some other spots, e.g. the notation describing recurrent network in Section 3 mentions an x_t which is surely different from the x_t used in the previous paragraph on regular feedforward NN*s, i think it belonged in the equation for h_t; the use of the two matrices A and P in Eq2 is strange, etc). Also, while Section 4.2 (Intuition for 2-cluster case) was a good idea to include and helpful, and while the *concepts* underlying the complexity analysis were straightforward, it could be made a lot clearer by (a) adding an additional figure such as Figure 2, along with (b) a few well-placed additional sentences unpacking the logic of the argument into easier-to-follow steps. For example, it was only when I saw Eq (6) and (7) combined with Fig(2) that the analysis on the previous page made more sense in terms of arriving at the eq for the complexity of putting the head of the distribution in the root of the tree. (Perhaps an Appendix might be the most appropriate place to add such an explanation).