Log-linear models models are widely used in machine learning, and in particular are ubiquitous in deep learning architectures in the form of the softmax. While exact inference and learning of these requires linear time, it can be done approximately in sub-linear time with strong concentrations guarantees. In this work, we present LSH Softmax, a method to perform sub-linear learning and inference of the softmax layer in the deep learning setting. Our method relies on the popular Locality-Sensitive Hashing to build a well-concentrated gradient estimator, using nearest neighbors and uniform samples. We also present an inference scheme in sub-linear time for LSH Softmax using the Gumbel distribution. On language modeling, we show that Recurrent Neural Networks trained with LSH Softmax perform on-par with computing the exact softmax while requiring sub-linear computations.
In this paper, the authors propose a new approximation of the softmax, based on approximate nearest neighbors search and sampling.
More precisely, they propose to approximate to partition function (which is the bottleneck to compute the softmax and its gradient), by using:
- the top-k classes (retrieved using LSH) ;
- uniform samples (to account for the tail of the distribution).
They describe how this technique can be used for learning, by performing sparse updates for the gradient (corresponding to the elements used to compute the partition function), and re-hashing the updated element of the softmax layers.
In section 5, they show how this method can be implemented on GPU, using standard operations available in neural networks framework such as TensorFlow or PyTorch.
Finally, they compare their approach to importance sampling and negative sampling, using language modeling as a benchmark.
They use 3 standards datasets to perform the evaluations: penn treebank, text8 and wikitext-2.
Pros:
 - well written and easy to read paper
 - interesting theoretical guarantees of the approximation
Cons:
 - a bit incremental
 - weak empirical evaluations
 - no support for the claim of efficient GPU implementation
== Incremental ==
While the theoretical justification of the methods are interesting, these are not a contribution of the paper (but of previous work by Mussmann et al.).
In fact, the main contribution of this paper is to show how to apply the technique of Mussmann et al. in the setup of neural network.
The main difference with Mussmann et al. is the necessity of re-hashing the updated elements of the softmax at each step.
Other previous works have also proposed to use LSH to speed up computations in neural network, but are not discussed in the paper (see list of references).
== Weak evaluations ==
I believe that the empirical evaluation of section 6 are a bit weak.
First, there is a large gap between the perplexity obtained using the proposed method and the exact softmax (e.g. 97 v.s. 83 on ptb, 115 v.s. 95 on wikitext-2).
Thus, I do not believe that the experiments support the claim that the proposed method "perform on-par with computing the exact softmax".
Moreover, these numbers are pretty far from what other papers have reported on these datasets with similar models (I am wondering if the gap would be even larger with SOTA models).
Second, the authors do not report any runtime numbers for their method and the baselines on GPUs.
I believe that it would be more fair to plot the learning curves (Fig. 1) using the runtime instead of the number of epochs.
== Efficient implementation ==
In section 5, the authors claims that their approach can be efficiently implemented on GPUs.
However, several of the operations used by their approach are inefficient, especially when using mini-batches.
The authors state that only step 2 is inefficient, but I also believe that step 3 is (compared to sampling approaches).
Indeed, for their method, each example of a mini-batch uses a different set of elements to approximate the partition function (while for other sampling methods, the same set is used for the whole batch).
Thus a matrix-matrix multiplication is replaced by n matrix-vector multiplication (n is the batch size).
While these can be performed in parallel, it is much less efficient than a matrix-matrix multiplication.
Finally, the only runtime numbers provided by the authors comparing their approach to sampling is for a CPU implementation with a batch of size 1.
This setting is super favorable to their approach, but a bit unrealistic for most practical settings.
== Missing references ==
Scalable and Sustainable Deep Learning via Randomized Hashing
Ryan Spring, Anshumali Shrivastava
A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models
Ryan Spring, Anshumali Shrivastava
Deep networks with large output spaces
Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga & Jay Yagnik