Joint quantile regression in vector-valued RKHSs
Addressing the will to give a more complete picture than an average relationship provided by standard regression, a novel framework for estimating and predicting simultaneously several conditional quantiles is introduced. The proposed methodology leverages kernel-based multi-task learning to curb the embarrassing phenomenon of quantile crossing, with a one-step estimation procedure and no postprocessing. Moreover, this framework comes along with theoretical guarantees and an efficient coordinate descent learning algorithm. Numerical experiments on benchmark and real datasets highlight the enhancements of our approach regarding the prediction error, the crossing occurrences and the training time.
The authors consider the problem of quantile regression. That is, given pairs (X,Y), the problem is to estimate specified quantiles of the conditional distributions Y|X=x in contrast to 'normal' regression in which the expectation E(Y|X=x) is to be estimated. Previous papers have phrased this as a regression problem for which functions from an RKHS with norm less than some constant are the hypothesis set.  The main contribution of this paper is to propose a method to tackle the so-called 'crossing problem', an issue that arises when curves for multiple quantiles are estimated simultaneously and the resulting curves cross over. The method the authors propose is to consider the problem as regression in a vector-valued RKHS. The kernel of the RKHS can be chosen in order to penalise the coordinate-wise functions crossing over.   They prove two theoretical results concerning the empirical risk minimiser of their problem. First, that the generalisation error is small with high probability; second, that the error in estimating the quantiles is small with high probability.   They write down the problem as an optimisation problem, and solve this using Primal-Dual Coordinate Descent.   Finally, they test their method on real datasets, showing that their method performs favorably compared to previous methods.
Novelty/Originality: The idea to apply vector-valued RKHSs to the problem of quantile regression appears to be novel. I hadn't encountered vector-valued RKHSs before reading this, but I don't know that much about this area. In any case, I like the idea because it is simple but appears to be effective. Although the authors restrict themselves to considering decomposable kernels, I would be interested to hear some discussion of other matrix-valued kernels and the resulting regression functions they would induce.  Technical Quality: The authors have provided two theoretical results about their method. I would appreciate more explanation of Theorem 4.2 (quantile property) as the statement of the theorem is a bit difficult to parse. I'd also appreciate some more discussion on selection of the parameter gamma - it seems like it should be interpretable in some way. At the end of section 6.1, it is stated that the pinball loss embraces the crossing loss as a subcriterion. I can see intuitively that this makes sense, but I'm confused as to why the crossing loss would then sometimes be so bad with the previous methods, even though they were also cross-validated using the pinball loss.  The following view might be unfashionable, but I would like to see the method being tested on synthetic data for which we know the conditional distributions Y|X=x and so can compare the estimated quantiles to the true quantiles. in addition to the real data experiments.  Potential impact: While I am not aware of any particular uses of quantile regression, it seems like it ought to be something that is practically used in the application areas listen in the introduction.    Clarity and presentation: In general, I thought that the section layout of the paper was good and the logic was clear. However, the style of writing was at times quite casual and there were a few grammatical errors.