Top Rank Optimization in Linear Time
Bipartite ranking aims to learn a real-valued ranking function that orders positive instances before negative instances. Recent efforts of bipartite ranking are focused on optimizing ranking accuracy at the top of the ranked list. Most existing approaches are either to optimize task specific metrics or to extend the rank loss by emphasizing more on the error associated with the top ranked instances, leading to a high computational cost that is super-linear in the number of training instances. We propose a highly efficient approach, titled TopPush, for optimizing accuracy at the top that has computational complexity linear in the number of training instances. We present a novel analysis that bounds the generalization error for the top ranked instances for the proposed approach. Empirical study shows that the proposed approach is highly competitive to the state-of-the-art approaches and is 10-100 times faster.
The paper addresses the computational issue of bipartite ranking. The authors propose a new algorithm whose computational complexity is linear in number of training instances, and provide theoretical analysis of generalization error. The paper is rounded of with extensive experiments.Strong points- The paper is clearly written and might be of some value in case of bipartite ranking with large datasets. The generalization bound is novel and experiments section is detailed.However, i have multiple questions about this paper:1. Though bipartite ranking is well studied, it is restricted in scope, in the sense that it is in the domain of ranking but cannot handle queries. Considering there are already well established algorithms for bipartite ranking which have been well studied theoretically and tested empirically, is the study really very valuable? For eg., this will only be useful when m,n are really large. Is that very practical in domain of bipartite ranking? Admittedly, this is just my thought and i would like to hear authors' view on this (citing example or something).2. The main reason the paper gets an O(m+n/\sqrt(e)) error bound is because of the new defined target loss (2) and using a smooth convex surrogate which allows standard primal-dual trick to get a smooth dual, thereby allowing standard accelerated gradient descent optimization. According to target loss (2), a negative instance followed by all positive instances has greater loss than a positive instance followed by all negative instances. Can this be considered promoting good ranking at top, especially since there is no position based discount factor? Moreover, in the supplement it is stated that maximizing positives at top cannot be achieved by AUC optimization, but target loss (2) is an upper bound on rank loss. So why should someone try to optimize an upper bound on rank loss, if AUC optimization itself is not suitable for the purpose of pushing positives at top?  3. Once the convex surrogate is taken to be smooth, conversion to dual and applying Nestrov technique is neat but i do not think it is extremely novel.4. Looking at empirical section, i am confused as to what TopPush is gaining on LR (logistic regression). Computational power of the new algorithm is the USP of the paper; it does not seem to be doing any better than LR; infact LR takes less time than TopPush in  4/7 experiments. Nor is it gaining anything significant in AP and NDCG metrics, effectively metrics which are popular; it gains a little in position at top but loses in AUC.  So why should we consider TopPush over LR? Or am i reading the experimental results wrong?A side point: On comparing the computational complexity with SVM (Rank,MAP); it can be seen that all of them scale linearly with training data size. The gain in computation time in TopPush is because SVM consider hinge loss while TopPush considers a smooth surrogate. So computational complexity linear in number of training instances is not unique to TopPush.5. Theory- In the generalization bound, shouldn't the focus be on the cases when there are a large number of negative instances and few positive instances? The other way round is less practical and even an average ranking function would put a few positive instances on the top. However, if we focus on the negative instances, i am not sure what the bound is relaying. With growing n, the empirical loss is much more likely to keep increasing, since the normalizing factor is only 1/m (no dependence on n). Since \delta will become small, the L.H.S probability is likely to grow but the R.H.S is also likely to grow. Maybe i am not being able to understand the significance of the bound, from a more useful n >> m point of view.   My ratings later on are provisional. I would like the authors to address the questions i have raised. Specifically i would like the authors to shed more light on the comparison between TopPush and LR ( 4, question on empirical section). In my opinion, the accept/reject hinges on clarifying how TopPush gains on LR. I will be happy to review my decision after author feedback.Update after Author Feedback- 1. "Example of m,n large"- I am not an expert in bipartite ranking, so i will take the authors' words for it. However, from my knowledge of online advertisement, is it not the case that ranking online advertisements is in the learning to rank framework? (i.e query dependent?). I completely agree with the first reviewer that showing experiments for large datasets (possibly real datasets used in bi-partite ranking) will be very useful.2. "AUC optimization"- The authors dont really answer the question. They talk about advantage of the new loss, in terms of optimization and gen. bounds. However, it has nothing to do with AUC. In fact, independent of how the new loss compares with AUC, the advantages will hold. From the point of view that the new loss is an upper bound on AUC and AUC is not useful for the objective of "pushing positives at top", why should someone optimize the new loss?3."comparison with LR"- This is critical. I agree that TopPush is doing better than LR in Pos@Top metric. I think in the revised version, the authors should modify the introduction slightly. The USP of the paper is the computational advantage of TopPush over other algorithms. This is overselling the paper a little bit. TopPush has no (visible) computational advantage over LR. It can be seen as an alternate, with advantage when it comes to performance on a specific metric (and disadvantage on some others). 4."SVM"- I believe the advantage over SVM based methods is the quadratic convergence rate (O(1/T^2) as opposed to O(1/T)), not linear in "m+n"?  Both SVM based methods and top push are linear in "m+n", as the authors have clearly shown in Table 1.5."Gen bound"- Please include the discussion in the revised draft. This is critical. Overall, i like this paper. With revision, it will certainly be a very good paper. I have updated my decision to an accept.