Efficient Optimization for Average Precision SVM
The accuracy of information retrieval systems is often measured using average precision (AP). Given a set of positive (relevant) and negative (non-relevant) samples, the parameters of a retrieval system can be estimated using the AP-SVM framework, which minimizes a regularized convex upper bound on the empirical AP loss. However, the high computational complexity of loss-augmented inference, which is required for learning an AP-SVM, prohibits its use with large training datasets. To alleviate this deficiency, we propose three complementary approaches. The first approach guarantees an asymptotic decrease in the computational complexity of loss-augmented inference by exploiting the problem structure. The second approach takes advantage of the fact that we do not require a full ranking during loss-augmented inference. This helps us to avoid the expensive step of sorting the negative samples according to their individual scores. The third approach approximates the AP loss over all samples by the AP loss over difficult samples (for example, those that are incorrectly classified by a binary SVM), while ensuring the correct classification of the remaining samples. Using the PAS-CAL VOC action classification and object detection datasets, we show that our approaches provide significant speed-ups during training without degrading the test accuracy of AP-SVM.
This paper proposes three different approaches to speed up the learning for AP-SVM (a structured SVM with AP loss).1. The first approach uses the unimodal behavior of the loss function, so uses binary search to speed up calculating the loss.2. The second approach notices that usually we have more negative samples compared to positive ones, and in calculating the loss, relative rank of negative samples falling between two consequent positive ones is not important. Hence, it speeds up by identifying all negatives between two consequent positive ones and assigns the same rank to all of them.3. finally, it approximates the training by using 0-1 loss for easy examples (easy under 0-1 loss SVM).In my view, all three tricks are interesting, clever, and effective in speeding up the training. They are described clearly. In working with big data, speeding up the training is an important problem, so NIPS audience are interested in this topic.I think the evaluation can be more thorough. Particularly, it might be important to try the algorithm on a different problem and also with larger number of training data. For instance, as I understood, the number of negatives in this task is not that large. There are other tasks like object detection that any possible bounding box in the image is considered a negative sample, so the number of negative samples can be exponentially large. I would suggest applying this algorithm on one of those tasks to show off the real power of this algorithm.In Table 1, I was thinking AP-SVM-SELECT should be faster than AP-SVM-SEARCH, but it is not the case. It might be the case in another experiment with larger number of examples. Also, in Fig 2-right, I was thinking by increasing the number of positive examples, AP-SVM-SELECT should loose its advantage and become slower than AP-SVM-SEARCH since the gaps become small, but it is not the case in this Fig.In Fig 2 middle, I am surprised that the binary SVM is slower than the AP-SVM-SEARCH. I thought the only difference is in calculating the loss, and calculating 0-1 loss is trivial. Maybe I didn't understand it completely.I think these should be discussed and explained if possible in the text.Is it possible to combine the first two tricks? Or even combine all three? What do we expect to see? Line 21 in abstract says they are complementary, so should be nice to show it.Minor points:Line 431, "hard" should change to "easy" to be consistent with Line 344. Am I right?Line 334, in "is less than 1", I think "1" should change to "-1".