Fast Prediction for Large-Scale Kernel Machines
Kernel machines such as kernel SVM and kernel ridge regression usually construct high quality models; however, their use in real-world applications remains limited due to the high prediction cost. In this paper, we present two novel insights for improving the prediction efficiency of kernel machines. First, we show that by adding "pseudo landmark points" to the classical Nyström kernel approximation in an elegant way, we can significantly reduce the prediction error without much additional prediction cost. Second, we provide a new theoretical analysis on bounding the error of the solution computed by using Nyström kernel approximation method, and show that the error is related to the weighted kmeans objective function where the weights are given by the model computed from the original kernel. This theoretical insight suggests a new landmark point selection technique for the situation where we have knowledge of the original model. Based on these two insights, we provide a divide-and-conquer framework for improving the prediction speed. First, we divide the whole problem into smaller local subproblems to reduce the problem size. In the second phase, we develop a kernel approximation based fast prediction approach within each subproblem. We apply our algorithm to real world large-scale classification and regression datasets, and show that the proposed algorithm is consistently and significantly better than other competitors. For example, on the Covertype classification problem, in terms of prediction time, our algorithm achieves more than 10000 times speedup over the full kernel SVM, and a two-fold speedup over the state-of-the-art LDKL approach , while obtaining much higher prediction accuracy than LDKL (95.2% vs. 89.53%).
The paper addresses the problem of speeding up prediction of kernel methods for large-scale problems for which the models are too large and hence the prediction is too slow. The proposed approach entails construction of pseudo landmark points, in the spirit of Nystrom approximation. Unlike the true landmark points which are sampled from the data, for pseudo landmark points the values of their kernel dot products with arbitrary points can be predicted from the same values for the landmark points. Hence the increased accuracy of model approximation can be attained with the same computational complexity as using the original set of landmark points. While the specific idea proposed in the paper is certainly novel, the overall conceptual motivation for the paper is somewhat weak in my opinion. Especially for large-scale problems, it is not the prediction but the training phase that constitutes the main computational bottleneck. It is virtually unfeasible to train large-scale models with nonlinear kernels when the kernel matrix must be stored in memory. Assuming this problem can be solved by low-rank approximation the complexity of the resulting model would then be tractable, hence I see no major gain to be made from speeding up the prediction phase. The idea of pseudo landmark points may be interesting to increase the prediction accuracy at a fixed complexity of the training model. The goal would then be not to accurately approximate the kernel matrix but rather to raise the predictive ability of the model (at fixed cost). However, the analysis provided in the paper does not go into this direction. On the technical part, several issues are unclear:1. No bias is considered in the SVM formulation in line 111. 2. Eq. 2: Why is G used in the definition of \bar{W}? This is counter-intuitive: we are trying to approximate G with \bar{G} depending on \bar{W} depending on G. A typo? 3. What is measured in Table 1? What is the meaning of 'x' in the values like '12.8x' etc? This table is not explained very well in the ensuing text. One can guess that the numbers pertain to some speedup, but what is the baseline for this comparison?  UPDATE:The notation used in the paper you have referred to in the response is much more clear, and computational implications are clearly explained. Please update your notation accordingly for the sake of self-sufficiency. The 'x' notation should be clearly expalained: I have almost misinterpreted it again when reading your response.