In information retrieval, learning to rank constructs a machine-based ranking model which given a query, sorts the search results by their degree of relevance or importance to the query. Neural networks have been successfully applied to this problem, and in this paper, we propose an attention-based deep neural network which better incorporates different embeddings of the queries and search results with an attention-based mechanism. This model also applies a decoder mechanism to learn the ranks of the search results in a listwise fashion. The embeddings are trained with convolutional neural networks or the word2vec model. We demonstrate the performance of this model with image retrieval and text querying data sets.
In this paper, the authors propose to use attention to combine multiple input representations for both query and search results in the learning to rank task. When these representations are embeddings from differentiable functions, they can be jointly learned with the neural network which predicts rankings. A limited set of experiments suggest the proposed approach very mildly outperforms benchmark approaches.
Major comments
To the best of my knowledge, this is the first paper to apply attention to the learning to rank problem. However, the main methodological innovation seems to be the use of attention to create and train an ensemble of models; this has been previously explored in the literature (e.g., [Kim et al., ECCV 2018]).
The paper is also missing important context in that it omits developments in using deep learning for the learning to rank problem (e.g., [Pang et al., CIKM 2017; Ai et al., WWW 2018]). The experimental evaluation does not include any other deep methods; thus, it is not clear if the (very minor) improvement in performance are due to the deep models or the proposed attention approach.
The datasets used in the experiments are not appropriate for evaluating learning to rank algorithms. A variety of learning to rank datasets are available, and these should be used rather than (or in addition to) the toy datasets considered here. Examples: http://arogozhnikov.github.io/2015/06/26/learning-to-rank-software-datasets.html, http://quickrank.isti.cnr.it/istella-dataset/, https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/, 
Minor comments
Concerning Section 3.3, in what sense is SGD used to “calibrate” the model? It seems as though the authors just mean it is used to “train” the model. However, is there some other meaning of calibration (e.g., in the sense of a Brier score) here?
In Table 1, what is the meaning of a dropout p value of 1? In most deep learning frameworks (e.g., Keras and PyTorch), this would mean all nodes are dropped out.
In what sense are the “5 randomized runs” for the experiments randomized? Are different train, test splits used? or just different random seeds? or something else?
How is it that the error rates are higher when using superclasses for evaluation?
Typos, etc.
The paper has several significant problems with the “\cite”s and “\ref”s in the paper. First, the “\cite”s should presumably be “\citep”s or something since the references are not set off from the rest of the text. Second, the paper includes references to equation numbers which are not present in the paper, such as “equation (12)”. It seems as the equations are in the paper, but are included in some unnumbered environment (“\begin{align*}” or some such). This makes it very difficult to track down to which equations the authors intend to refer. Third, the reference numbers to figures and tables in the text is wrong. For example, the text refers to “Tables 8 and 9” for 20 newsgroups (at the end of Section 4). Clearly, this is supposed to be Tables 6 and 7. It seems like the authors moved the CIFAR-10 discussion to the appendix but did not update the references in the text.
Tables 2 and 4 are exactly the same.
Figure 4 is not referenced in the text.
It would be helpful to put Figure 2 a bit closer to where it is discussed in the text.
The references are not consistently formatted.
“components of for each” -> “components for each”
Please define acronyms like MAP at least once.