One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities
The softmax representation of probabilities for categorical variables plays a prominent role in modern machine learning with numerous applications in areas such as large scale classification, neural language modeling and recommendation systems. However, softmax estimation is very expensive for large scale inference because of the high cost associated with computing the normalizing constant. Here, we introduce an efficient approximation to softmax probabilities which takes the form of a rigorous lower bound on the exact probability. This bound is expressed as a product over pairwise probabilities and it leads to scalable estimation based on stochastic optimization. It allows us to perform doubly stochastic estimation by subsampling both training instances and class labels. We show that the new bound has interesting theoretical properties and we demonstrate its use in classification problems.
The authors introduce a lower bound on the softmax probability frequently used in multi-class learning, based on combining one-vs-one probabilities. They demonstrate that this bound can be used to train large-scale multi-class models.
In my view, the main reason the proposed lower bound is interesting is that it offers a potential way to speed up training for multi-class models with a very large number of classes. While it is useful to understand other properties of the lower bound, the paper could be improved by emphasizing this primary use case in machine learning.  Figure 1c and Figure 3 need a more clear explanation of what is being displayed, and why it is important. In particular, what value is being plotted on the y-axis, and at what setting of the parameters w. Here is how I understand it, for Figure 1c say:  Blue Line - value of Eq. (13) at the setting of parameters w that maximize 13  Red Line - value of Eq. (13) at the setting of parameters w that maximize 14  Green Line - value of Eq. (13)? at the setting of parameters w that maximize the Bouchard lower bound (?) Red dashed line - value of Eq. (13)? at parameters w based on the given iterations of training? But if it is always the actual log-likelihood on the y axis (Eq 13), then I'm not sure why the y-axis is labelled "lower bound" and you use the phrase "maximized (approximate) log likelihoods". The same comments apply to Figure 3.  In order to effectively demonstrate that the proposed bound can significantly speed up training, additional experiments and comparisons are needed. Most importantly, another natural proxy loss that can be used in training is to only compute the softmax over a random sample of the full output space (vocabulary.) I believe this is a fairly standard trick (e.g., implemented in tensorflow as tf.nn.sampled_softmax_loss, https://www.tensorflow.org/versions/r0.9/api_docs/python/nn.html#sampled_softmax_loss). See also Section 3 of http://arxiv.org/pdf/1412.2007v2.pdf and references therein.  A critical parameter in both approaches is the number of classes used in the approximation of the softmax. I'd like to see that parameter explored, with some guidance on how to choose it in practice. For example, in Figure 2(c) I'd like to see lines for the number of classes in the stochastic approximation ranging from 1 to 10^4, as well as a line for SGD with computation of the full softmax. But ideally this parameter would be explored on the real datasets as well, with the key metric being wall-clock-time to reach a given level of accuracy.  For the small-scale experiments, in Table 2, the Softmax column has its fields in a different order, and is (critically) missing the error column. It's not clear from the text if the metrics are on the training set or the test set --- I assume the test set. But in that case it's hard to dismiss the Bouchard method as overfitting if it is actually getting the lowest test-set error. This method is more computationally expensive, but if that is the only downside, you should include experiments that show this (in particular to justify not including it in the comparison on the large-scale problem).  For the amazon-13k results, no comparisons are given, so it is hard to draw conclusions. To be meaningful, there should be a comparison to the sampled softmax mentioned above. You could also compare to full softmax but with a random projection or hashing trick on the input features to a lower-dim space (few thousands) where this would be more practical --- this would be a natural baseline. Again, since computational speedups are the main potential advantage here, those should be clearly illustrated.  Minor comments: Figure 1(a): I assume these are probabilities estimated by optimizing using the given loss, but then with probabilities all computed using Eq (2) (rather than say computing estimated probabilities for the Bouchard method using Eq. (11)). Please clarify this. Several typos and missing words, e.g. line 10 (allows *us*), 37 (*wheneven*), 114 (Not only *is F(f)* globally …), 122 (in order *for* the bound). Line 244 should refer to Table 2 not Table 1. Lines 235-237: These methods are all naturally singly stochastic, and can (and probably should) be solved with SGD (sampling examples) not GD. It should be clarified that while the global optimum is the same for Eqs. (6) and (8), it is not the case that (13) and (14) share the same minimum.