The objective in deep extreme multi-label learning is to jointly learn feature representations and classifiers to automatically tag data points with the most relevant subset of labels from an extremely large label set. Unfortunately, state-of-theart deep extreme classifiers are either not scalable or inaccurate for short text documents. This paper develops the DeepXML algorithm which addresses both limitations by introducing a novel architecture that splits training of head and tail labels. DeepXML increases accuracy by (a) learning word embeddings on head labels and transferring them through a novel residual connection to data impoverished tail labels; (b) increasing the amount of negative training data available by extending state-of-the-art negative sub-sampling techniques; and (c) re-ranking the set of predicted labels to eliminate the hardest negatives for the original classifier. All of these contributions are implemented efficiently by extending the highly scalable Slice algorithm for pretrained embeddings to learn the proposed DeepXML architecture. As a result, DeepXML could efficiently scale to problems involving millions of labels that were beyond the pale of state-of-the-art deep extreme classifiers as it could be more than 10x faster at training than XML-CNN and AttentionXML. At the same time, DeepXML was also empirically determined to be up to 19% more accurate than leading techniques for matching search engine queries to advertiser bid phrases. Source code for DeepXML can be downloaded from (Anonymous, 2019).
This paper considers extremely multi-label learning (XML) where the label size is very large . It aims to improve the accuracy as well as the scalability of XML algorithms, especially for short text inputs. The accuracy for XML with short text inputs can be significantly improved using deep learning representation than using TFIDF features. This paper proposes several tricks to handle the issue for efficiently learning both neural network parameters and classification weights of extremely large number of labels. The proposed method borrowed ideas from several previous literature, and is mainly based on SLICE, where a pre-trained fixed deep learning representation for the inputs are used with ANNS graph for labels to improve the scalability. The main difference is that instead of using a fixed input embedding, the proposed method learns the word embedding parameters via a set of head labels. The remaining labels are then trained using SLICE with fixed word embeddings from the learned word embedding model. 
Overall the paper tackles the problem well. And the empirical results show improved results. However, I don't think this paper is ready for publication due to the following concerns. 
1. My main concern is that the proposed method seems to be a combination of a number of tricks. This makes the overall algorithm/model very complicated and introduces a lot of hyper-parameters, for example, head label portion, L-h', c, beta, s neural network hyper-parameters and so on. Hence, it will be hard to be used in real applications.  
2. Another concern is about the experiments. 
    a. The most significant improvement of the proposed method over existing method happens in the private dataset, Q2B-3M, which can't be reproduced.  
     b. On the public datasets, DeepXML seems to show good results on small datasets, WikiSeeAlsoTItles-350K and WkipediaTitle-500K, while on large datasets, DeepXML performance is close to the existing methods. 
     c. The largest label size in the experiments is 3M. SLICE can be scaled up to 100M labels. 
3. The writing and the organization of this paper needs to be improved. 
    a. Some notations are not clearly defined. For example, L_h in Line 6 and X' in Line 9 on Page 5. 
    b. Several method names are not defined. For example, DeepXML-fr, AttentionXML-l, Sliced-CDSSM, DeepXML-SW, DeepXML-f. I have to guess what they are. 
    c. The last two paragraphs on Page 4 seems to be related work, while there is a section called "Related work".
Other minor comments:
1. It seems it is not stated how Beta is set. 
2. I am wondering if it's true that the shorter the input text is, the better improvement over non-deep-learning methods DeepXML can achieve.
3. In the first paragraph of Sec 3.1, it is mentioned "clustering just the top 3 labels into 300 clustering". Why choose 3 and 300? Are these numbers used for all datasets?