Less is More: Nyström Computational Regularization
We study Nyström type subsampling approaches to large scale kernel methods, and prove learning bounds in the statistical learning setting, where random sampling and high probability estimates are considered. In particular, we prove that these approaches can achieve optimal learning bounds, provided the subsampling level is suitably chosen. These results suggest a simple incremental variant of Nyström Kernel Regularized Least Squares, where the subsampling level implements a form of computational regularization, in the sense that it controls at the same time regularization and computations. Extensive experimental analysis shows that the considered approach achieves state of the art performances on benchmark large scale datasets.
Summary: The authors study the computational/statistical tradeoff of Nystrom methods in the kernel ridge regression problem in case of random design. They derive optimal finite sample bounds for the excess risk of the estimator (Theorem 1) by choosing the (n,m,\lamda) [n - number of samples, m - number of subsamples in the Nystrom method, \lambda -regularization parameter] appropriately. The paper also gives insight how 'm' plays the role of regularization both from theoretical (computational error term) and practical point of view (numerical experiments). The proof relies on approximation theory and functional analysis; the combination of these results to obtain the bounds is nice and technical. The paper is well organized, the results are clearly presented and relevant.Below I enlist my major comments, improvement suggestions; my minor comments are listed afterwards:-Generally, I would encourage the authors to avoid the 'it is easy to see' type of statements: such _conjectures_ are often error-prone (some additional derivation never hurts).main: -line 280-281: In '1)' is the second term really of the same order as the first and third one? This requirement would imply two (possibly contradictory) equations for 'm'. Please detail and clarify the statement. -line 338-342: The stated update rule via the block matrix inversion lemma is screwed up (in \alpha_t: M_t:(tl)x(tl), C_t:nx(tl), \hat{y}_n: nx1 -- they are not compatible; C_1 is undefined; in D_t, the last B^t could be B_t, ...). Please, provide a precise derivation and formula.supp: -Lemma 3: In the proof of Lemma 3 Prop. 11 is applied, which requires H=H(K) to be a _separable_ Hilbert space. An RKHS is not necessarily separable; thus, assume/guarantee it. A relatively mild sufficient condition is 'separable topological domain with a continuous kernel'; see Lemma 4.33 in [25]. -Lemma 1: --\alpha comes from line 559, thus in line 555 \alpha should be \tilde{\alpha} and \hat{y} should be \hat{y}_n [in accord with Eq. (12)]. From the first term (=pseudo-inverse) an 1/n multiplier appears, from B_{mn} an \sqrt{n} => an \sqrt{n} term seems to be missing from the final expression. ? --line 549: This \hat{y}/\hat{y}_n definition is unclear: \hat{y}_n has already been defined (line 98)... -line 858: Based on the B1 bound, the exponent of C_m in Eq. (15) seems superfluous (i.e., it is equal to 1); the error propagates to line 928 and 963.Minor comments/typos: ==================== main: -||operator||: define/denote that this is the operator norm (not Hilbert-Schmidt norm/...); the same holds for vector norms (it appears in the text as ||v|| and ||v||_{H} as well; I would recommend to use the 2nd form) -line 88: \lambda \ge 0 -> \lambda > 0 (otherwise for example uniqueness might not hold) -line 113: The (K_{mn})_{ji} = K(\tilde{x}_j,x_i) (or K_{mn}=K{nm}^T) definition is missing -line 267: \hat{f}_{\lambda m} -> \hat{f}_{\lambda,m} (comma is missing, see also throughout the paper) -line 280: approah -> approach -line 319: Eq.5 -> Eq.~(5); see also line 374. -line 319: the selected points -> be the selected points -line 320: dimension l -> size l -line 320: please add "t=1,...,T" -line 338: \alpha_t should be \tilde{\alpha}_t -line 344: to compute \tilde{a}_1,...,\tilde{a}_T -> to compute \tilde{\alpha}_1,...,\tilde{\alpha}_T -References: [9,14,18]: the accents are missing from Nystromsupp: -line 515: the this -> this -line 532: ']H' -> _{H} -line 533: Z_m* (the '*' is missing on the left) -line 541: bold K -> K_n (see under Eq. (4)) -line 541: "B_{nm} = \sqrt{nm}S_n Z_m*" -> "B_{nm} = \sqrt{n}S_n Z_m*" ('m' seems to be superflous) -line 541: "G_{mm} = m Z_m Z_m*" -> "G_{mm} = Z_m Z_m*" (m is not needed) -line 542: bold \tilde{K} -> \tilde{K}_n [see Eq. (13)] -line 571: UU* should be U*U (it occurs twice) -line 597: S_n should be S_n*; the same holds in line 604, 861 -line 622: \sqrt{()^2} is superfluous -line 636-638: partition -> subset (twice); see also line 650 -line 644: C_m should be "H -> H (or H_m)" type mapping, not "H -> R^m" -line 646: C_{\lambda} has not been defined (only later) -line 663: the the -> the -line 719: Therfore -> Therefore -line 725: colum -> column -line 855: the accent is missing from 'Nystrom' -line 884: What is r? -line 895-896: The 's' character is already in use (see Assumption 4), please select an other one. -line 935: taking -> take-line 1050: "let denote || ||_HS" -> "let || ||_HS denote" -line 1052: "Let name" -> "Let"; the same holds in line 1063 -line 1087: "Let n, n ... integers" -> "Let n ... integer" -line 1099: ". For any" -> ", then for any" -Please add fullstops (for example) to line 578, 598, 632, 985, 997, and use commas in line 604, 858, ...