Fast Randomized Kernel Ridge Regression with Statistical Guarantees *
One approach to improving the running time of kernel-based methods is to build a small sketch of the kernel matrix and use it in lieu of the full matrix in the machine learning task of interest. Here, we describe a version of this approach that comes with running time guarantees as well as improved guarantees on its statistical performance. By extending the notion of statistical leverage scores to the setting of kernel ridge regression, we are able to identify a sampling distribution that reduces the size of the sketch (i.e., the required number of columns to be sampled) to the effective dimensionality of the problem. This latter quantity is often much smaller than previous bounds that depend on the maximal degrees of freedom. We give an empirical evidence supporting this fact. Our second contribution is to present a fast algorithm to quickly compute coarse approximations to these scores in time linear in the number of samples. More precisely, the running time of the algorithm is O(np 2 ) with p only depending on the trace of the kernel matrix and the regularization parameter. This is obtained via a variant of squared length sampling that we adapt to the kernel setting. Lastly, we discuss how this new notion of the leverage of a data point captures a fine notion of the difficulty of the learning problem.
This paper extends on previous work relating to the Nystrom method of low rank approximation of SPSD (e.g. kernel) matrices, showing that tighter bounds can be derived that the sharp analysis of Bach using a non-uniform sampling strategy that attempts to control the bias. The idea of non-uniform sampling for the Nystrom approximation is not itself new, but the analysis goes further than this, and holds for any sketching matrix that meets certain conditions. This, combined with the resultant bounds, is a significant contribution. Comments/criticisms: - the title is "Fast Randomized Kernel Methods ..." whereas currently the analysis only holds for Kernel Ridge Regression. **Note that the title was changed for final submission** - it seems that this method will not work for any normalised kernel matrix (i.e. where f(x) = \phi(x)/ \left| \phi(x) \right|), since the p_i's will be uniform. For example, the RBF kernel will always have 1's on the diagonal, so K_{ii} / TR(K) will just be 1/n? - in the first experiment, what is the MSE when using full KRR on this dataset? does the lambda ridge method converge to this error? - in the second experiment (Table 1), why haven't MSE's been reported? A comparison with existing methods should also be done here.Specific comments: - for ref [1] it would be better to refer to the published ICML paper than the unpublished arxiv paper unless there's a good reason not to (http://machinelearning.wustl.edu/mlpapers/bibtex/icml2013_gittens13) - for ref [5] same as above (http://jmlr.org/proceedings/papers/v32/ma14.html) - L67, L256 the authors refer to Bach's d_{mof} as "maximal degrees of freedom", but I believe Bach called this "marginal degrees of freedom" - eq 4 (and later) the first term is given as bias(K)^2 whereas Bach denoted this same term as simply bias(K) - is there a reason for changing this? - L158The operator  is used for the first time. I presume you are referring to the linear matrix inequality X \succeq Y \Longleftrightarrow \lambda_{min}(X-Y)\geq 0, but I'm not sure all readers will be aware of this - L162 derived of => derived from - Theorem 2. State what t is (i.e. t \in (0, 1)?) - L302-305 Section 3.5. The last sentence in the first paragraph is unclear and too long, please reword this - L373 experiments -> experiment - Figure 1. These figures are too small to be readable. I realise space is an issue but please make these more readable if possible. - L398 haven't we known there existence => had we not known of their existence - L399 determine \lambda => determine \lambda and \sigma - Table 1. \sigma was used earlier in the paper to refer to eigenvalues, now it refers to the RBF parameter - could a different symbol be used? - throughout the paper (and references inverted commas are the wrong way round, caused by using the double quote " character instead of two single quotes '' - In the references the capitalisation is wrong in many cases: conference/journal papers should have title case (except for proper nouns and abbreviations), whereas books should have every word capitalized. Nystrom should appear as Nystr\"om. Journal papers should have issue numbers and page numbers.