Ranking Data with Continuous Labels through Oriented Recursive Partitions
We formulate a supervised learning problem, referred to as continuous ranking, where a continuous real-valued label Y is assigned to an observable r.v. X taking its values in a feature space X and the goal is to order all possible observations x in X by means of a scoring function s : X → R so that s(X) and Y tend to increase or decrease together with highest probability. This problem generalizes bi/multi-partite ranking to a certain extent and the task of finding optimal scoring functions s(x) can be naturally cast as optimization of a dedicated functional criterion, called the IROC curve here, or as maximization of the Kendall τ related to the pair (s(X), Y ). From the theoretical side, we describe the optimal elements of this problem and provide statistical guarantees for empirical Kendall τ maximization under appropriate conditions for the class of scoring function candidates. We also propose a recursive statistical learning algorithm tailored to empirical IROC curve optimization and producing a piecewise constant scoring function that is fully described by an oriented binary tree. Preliminary numerical experiments highlight the difference in nature between regression and continuous ranking and provide strong empirical evidence of the performance of empirical optimizers of the criteria proposed.
The paper studied the problem of continuous ranking, which is a generalization the bi-partite/multi-partite ranking problem in that the ranking label Y is a continuous real-valued random variable. Mathematical formulation of the problem is given. The problem of continuous ranking considered as multiple sub-problems, each corresponds to a bipartite ranking problem. Necessary conditions for the existence of optimal solutions are given. The IROC (and IAUC) measures are proposed for evaluating the performances of scoring functions. Finally, a continuous ranking algorithm called Crank is proposed. Experimental results on toy data showed the effectiveness of the proposed Crank algorithm. 
The problem investigated in the paper is very interesting and important for learning to rank and related areas. The paper is well written. The proposed framework, measures, and algorithms are clearly presented. The empirical evaluation of the paper is weak, as it is based on toy data and weak baselines. 
Pros. 
1.	Generalizes the bipartite/multi-partite ranking problems to the continuous ranking problem. Rigorous formulation of the problem and strong theoretical results. 
2.	Extending the conventional ROC to IROC for measuring the continuous ranking functions. 
3.	Proposing Crank algorithm for conducting continuous ranking. 
Cons. 
1.	The authors claim that “theoretical analysis of this algorithm and its connection with approximation of IROC* are beyond the scope of this paper and will be the subject of a future work”. I am not convinced that “its connections with approximation of IROC* are beyond the scope of this paper”. As to my understanding of the paper, the goal of proposing IROC is to guide the proposal of new continuous ranking algorithms. Thus, it is very important to build the connections. Otherwise, the performances of the Crank algorithm cannot reflect the effectiveness of the proposed theoretical framework. 
2.	The experiments in the paper are really weak. The proposed Crank algorithm is tested based on a toy data. The authors conclude in the last section “… finds many applications (e.g., quality control, chemistry)…”. It is necessary to test the proposed framework and solutions on real problems. The baselines are CART and Kendall, which are not designed for the ranking problem. It is better to compare the algorithm with state-of-the-art bipartite and multi-partite ranking models. The generation of the toy examples, the setting of parameters are not given, which makes it is hard to reproduce the results. 
3.	The literature survey of the paper is not enough. Since the continuous ranking problem is a generalization of the bipartite ranking and multi-partite ranking problems, it is better if the authors could use a section to analyze the advantages and disadvantages of existing ranking models, and their real applications. Currently, only a few references are listed in Section 2.2. 
4.	Minor issue: Line 52: Kendall tau  Kendall $\tau$; Line 66: F^{-1}(u) = inf{s\in R: F(s}>= t), the last t should be u;