Training a classifier over a large number of classes, known as 'extreme classification', has become a topic of major interest with applications in technology, science, and e-commerce. Traditional softmax regression induces a gradient cost proportional to the number of classes C, which often is prohibitively expensive. A popular scalable softmax approximation relies on uniform negative sampling, which suffers from slow convergence due a poor signal-to-noise ratio. In this paper, we propose a simple training method for drastically enhancing the gradient signal by drawing negative samples from an adversarial model that mimicks the data distribution. Our contributions are three-fold: (i) an adversarial sampling mechanism that produces negative samples at a cost only logarithmic in C, thus still resulting in cheap gradient updates; (ii) a mathematical proof that this adversarial sampling minimizes the gradient variance while any bias due to non-uniform sampling can be removed; (iii) experimental results on large scale data sets that show a reduction of the training time by an order of magnitude relative to several competitive baselines.
The paper presents a method for negative sampling for softmax when dealing with classification of data to one from a large number of classes. Its main idea is to negative sample those classes which lead to higher signal to noise ratio than for uniform negative sampling. This is based on building an auxilary model using decision tree from which the adversarial negative classes are sampled, so that the distribution of the negative samples can be close to the positive ones leading to higher SNR while training. The proposed method is compared to other methods for negative samping on two publicly available large-scale datasets from the extreme classification with XML-XNN features. 
Positives :
1. The proposed approach with adversarial negative sampling using an auxilary model seems interesting 
2. It scales well to datasets with large number of classes.
Negatives :
The experimental evaluation of the proposed approach lacks completeness and does not look convincing for the following reasons : 
1. It misses out a recent state-of-the-art method (Slice) for negative sampling on same datasets [1], which also addresses the same problem of sampling most promising negative classes but in a different way. Furthermore, [1] also compares against many other sota methods missed out in this paper on the many other datasets datasets including those in this paper but in a more general multi-label setting.
2. The paper only compares against other negative sampling approaches such as AandR, NCE, and does not show what happens when no negative sampling is done such as done in (DiSMEC) [2]. This is important to understand what (if at all) is lost by doing approximation as proposed. For instance, a quick experiment reveals that DiSMEC can give about 19% accuracy on Wiki500 dataset, which is better than that achieved by the proposed method. Though it is computationally expensive but due to its simplicity, it must be discussed nevertheless to give a complete picture.
Instead the OVE baseline used in the paper seems quite sub-optimal in the first place, and hence stronger baselines [1,2] for which the code and results are readily available and have been duly tested in the community must be used and discussed.
Another aspect that the paper misses out is the role of fat-tailed distribution [3,4] of the instances among labels, which is a property of typical datasets in this regime. It is possible that one can get good accuracy but poor performance on tail-labels due to approaximations. The performance on tail-labels on appropriate metrics other than accuracy, such as MacroF1, should be evaluated.
Also, the proposed approach must be tested on more datasets including the smaller ones such as EURLex (also used in works referenced in the paper) on which it is easier to compare with other methods (such as DiSMEC, Slice and AttentionXML [5]) without encountering computational constraints and also bigger ones such as Amazon3M, also avilable from the repository. 
Finally, it must be investigated if the proposed method can be extended to the multi-label setting or are there inherent limitations of the model in this setting. The possibility to extend it to the general multi-label setting would make this approach more promising and directly comparable to wide range of algorithms.
[1] H. Jain,  V. Balasubramanian,  B. Chunduri and M. Varma, Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches, in WSDM 2019.
[2] R. Babbar, and B. Schölkopf, DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification in WSDM, 2017.
[3] H. Jain, Y. Prabhu, and M. Varma, Extreme Multi-label Loss Functions for Recommendation, Tagging, Ranking & Other Missing Label Applications in KDD, 2016.
[4] R. Babbar, and B. Schölkopf, Data Scarcity, Robustness and Extreme Multi-label Classification in Machine Learning Journal and European Conference on Machine Learning, 2019.
[5] AttentionXML: Extreme Multi-Label Text Classification with Multi-Label Attention Based Recurrent Neural Networks, NIPS 2019