Error Analysis of Generalized Nyström Kernel Regression
Nyström method has been successfully used to improve the computational efficiency of kernel ridge regression (KRR). Recently, theoretical analysis of Nyström KRR, including generalization bound and convergence rate, has been established based on reproducing kernel Hilbert space (RKHS) associated with the symmetric positive semi-definite kernel. However, in real world applications, RKHS is not always optimal and kernel function is not necessary to be symmetric or positive semi-definite. In this paper, we consider the generalized Nyström kernel regression (GNKR) with 2 coefficient regularization, where the kernel just requires the continuity and boundedness. Error analysis is provided to characterize its generalization performance and the column norm sampling strategy is introduced to construct the refined hypothesis space. In particular, the fast learning rate with polynomial decay is reached for the GNKR. Experimental analysis demonstrates the satisfactory performance of GNKR with the column norm sampling.
The authors focus on the kernel ridge regression task with l_2-coefficient regularization [Eq.~(3)], on a compact domain of R^d. The primary goal is to analyze the performance of the Nystrom technique in this task, with 'Mercer kernels' (continuous bounded functions), which are not necessarily positive symmetric or positive definite. The main contribution is a generalization error bound on Nystrom based estimator (Theorem 1) which can give rise to fast rates (Theorem 2) if the true regression function satisfies an exponential approximation requirement (Definition 2) with a C^\infty kernel. The authors numerically illustrate that in the studied task the Nystrom method with column-norm sampling typically performs better compared to Nystrom technique relying on uniform sampling.  The paper is 1)novel: it bridges the gap between two popular directions, l_2-coefficient regularized objective functions and approximation with the Nystrom scheme. 2)well-organized, clearly written.   There are a few statements in the paper which should be proved and some bugs to be corrected. I detail these comments, and my minor points in the 'Qualitative assessment' section.
I use the 'lX' = 'line X' shorthand below. 'X->Y' will mean "Please change 'X' to 'Y'".  Major comments: ==============  A)Main part: ----------- 1)I found the 'Introduction' section redundant: the issues and the contributions are quite similar, this part can be significantly compressed. Throughout the paper the notion of 'Mercer kernel' is left a bit undefined, even in the 'exact' definition ('line 96'); currently, it has to be decoded from fragments of the abstract and the introduction. I also suggest adding  i)more emphasis on the motivation of general kernels (not necessarily symmetric and positive definite): What are the properties (a few illustrative examples can always help) which are not captured by the  classical reproducing kernels?, further potential impact, ... ii)Nystrom alternatives in the introduction such as the random Fourier feature technique [1-5] or the sketching method [6].  References: [1] A. Rahimi and B. Recht. Random features for large-scale kernel machines. In NIPS-2007, p. 1177–1184. [2] D.J. Sutherland and J. Schneider. On the Error of Random Fourier Features. In UAI-2015, p. 862-871. [3] B. Sriperumbudur, Z. Szabo. Optimal Rates for Random Fourier Features. In NIPS-2015, p. 1144-1152. [4] F. Bach. On the equivalence between quadrature rules and random features. Technical Report, 2015. (http://arxiv.org/abs/1502.06800) [5] A. Rudi, R. Camoriano, L. Rosasco. Generalization Properties of Learning with Random Features. Technical Report, 2016. (https://arxiv.org/abs/1602.04474) [6] Y. Yang, M. Pilanci and M. J. Wainwright. Randomized Sketches for Kernels: Fast and Optimal Non-Parametric Regression. Annals of Statistics (to appear; http://arxiv.org/abs/1501.06195).   2)There are some statements/conjectures in the paper without proof: i)l103-106: Solution of (1). Note: the cited [18] paper does not include 'inf_{\alpha}' in the regularization term. In fact, without this 'inf_{\alpha}' the regularization is not even a function of \alpha (unless the k(\cdot,x_i) points are assumed to be linearly independent). ii)l110-112: Solution of (2). iii)l130-132: (F,||\cdot ||_F) is a Banach space. iv)l203: Equivalent form of (9). Please, add proofs for these statements.  3)The computational complexity of the p=(p_i) vector in the column-norm sampling scheme seems to be O(n^2). This should also increase the complexity of (4) from O(mn+m^3), shouldn't it?   4)Section 3-4: In these sections there are several function definitions (f_\rho, f_z, \pi(\f_z), f_\lambda, g_\lambda, \hat{g}_\lambda). Based on the main text it is not obvious how these quantities will be connected, i.e. what will be the bridges. If there is place left [there are many displayed equations which can be made inline, or collected (see for example H, ||g||_H) to save space], it would be great to give a few-lines-long proof idea in the main part of the paper, 'the intuition of the bridges'.  5)Could the authors provide some concrete/illustrative examples, easy-to-check sufficient conditions for f_\rho in Def. 2? It is nice show that actually there are problems where this assumption holds.  6)The interpretation of Theorem 2 could be made more accessible by a quick discussion on the value of p and its impact, such as "p\in (0,2), m = n^{1/(2+p)} means that m can be chosen between n^{1/4} and n^{1/2}". Similarly noting that 'The smoothness of K is encoded in Def. 2.' can help understanding.  7)l195-206: This part should be improved. The displayed equation is hard to interpret; one could write instead "S(p_1,\ldots,p_n) = ..., task: \min_{p_1,\ldots,p_n}S(p_1,\ldots,p_n).". In line 203 "L" is undefined. "h_ii" is undefined; based on the supplement it should be "L_ii". "the minimizer of (9) can be approximated by": from the main text it is not clear in which sense this holds. A short sentence/discussion on the intuition of the "L_ii->0" assumption is also useful to add.  B)Supplement: ------------- 1)Unproved statements: l25: \phi'(0) = 0.  l26-27: "according to the arbitrariness of h, we get \lambda f_\lambda(t)=...". Note: I suppose that the authors wanted to construct some (h_n) \subset \L^2_{\rho_X} sequence and apply a limiting argument. l42-43: last inequality in (2). l102-103: order of "\sqrt{\frac{\log(2/\delta)}{m \lambda}}", see 2) below. Lemma5: 'R can be large enough', see 3) below. Please prove these gaps.  2)l102-103: Check the constants of the 3rd term on the r.h.s. [=D(\lambda) x (...)]. An "\sqrt{\frac{\log(2/\delta)}{m \lambda}}" type term is missing (it is coming from the E_2 bound = l37-38); if it is dominated by some other term, justify its reason. The same '\sqrt{...}' comment holds for Theorem 2.  3)l94-95: It looks like an 'for R is large enough' assumption has been implicitly used here.   Note: By assumption D(\lambda) \le c_\beta \lambda^{\beta}, where \beta (0,1]. R = \sqrt{D(\lambda)}/lambda \le \sqrt{c_\beta \lambda^{\beta}} / \lambda is just an exploding upper bound as \lambda ->0. Show that 'R is large enough' for sufficiently small \lambda, or assume it in Theorems 1-2.  Minor points:  ============  A)Main part: ----------- 1)l96-97: The boundedness of y is assumed; please make this assumption explicit -- otherwise w.l.o.g. one can not have y \in [-1,1]. In fact, it would be nice to leave the general kernel bound (\kappa) and the bound on y to make it explicit how these constants appear in the final performance guarantee (in a linear/quadratic/... form). 2)l123: This part of the sentence reads a bit strange. 3)l127-128: 'Let F be a square integrable space on X...' -> 'Let F be the square integrable space on X...'. Note: For a subset of L^2_{\rho_X}, the range of L_K has to be studied separately. 4)l130-132: in the definition of H: 'g(\cdot)' -> 'g', 'f(\cdot)' -> 'f'. 5)l130-132: in the definition of ||\cdot||_H: 'g(x)' -> 'g', 'f(x)' -> 'f', 'x \in X' -> ''. 6)l143, l165: 'the generalization error bound' -> 'our generalization error bound', 'we provide the main results' -> 'we provide our main results' [reading the text it is not really explicit that these are the authors' results] 7)After Definition 2: a quick note could be useful on the meaning of '\beta', such as 'larger \beta means easier task'. 8)line 215: in the K_E definition: '(...)' -> '\left(...\right)'. 9)Section 5.1: The authors could refer back when listing f_i-s, that these are the true f_\rho-s. 10)l241: 'Figure 1 shows the performance of the three sampling strategies' -> 'Figure 1 shows the performance of the two sampling strategies' [3->2]. Note: the subfigures in Fig. 1 could be arranged to a 3X3 format to save space. 11)References: [1,2,8,14]: missing page numbers. [3]: 'monte carlo' -> 'Monte Carlo'. [6]: 'reguularization' -> 'regularization', 'bouunds' -> 'bounds'. [20,25]: missing url-s.  B)Supplement: ------------ l3: 'provides the detail proof' -> 'provides the detailed proof' l12-13: the inequality seems to hold with equality as well l22: 'E_z(0) = 1' -> 'E_z(0) \le 1' since |y_i| \le 1 [i.e., this equality should be inequality]. l23: 'a auxiliary function' -> 'an auxiliary function' l24: 'h is an any fixed function' : at least h\in L^2_{\rho_X} seems to be needed to make the integral well-defined l57: "N_{2,u}(F,\epsilon) = N_{2,u}(...)": on the r.h.s. the '2,u' subscript should be deleted (to be compatible with Def. 1), l62: "Ef \le c (Ef)^\alpha" -> "Ef^2 \le c (Ef)^\alpha", i.e. f should be squared on the l.h.s. l77: r.h.s. of |g_1(z)-g_2(z)| should be '| |2y-\pi(f_1)-f_\rho(x)| |\pi(f_1)(x)-f_\rho(x)| - |2y-\pi(f_2)-f_\rho(x)| |\pi(f_2)(x)-f_\rho(x)| |'. l80: "true for and g" -> "true for any g" l81: please add the "\alpha = 1" choice. l87: 'It is easy to check that \hat{g}_{\lambda} \in G' -> 'By the definition of \hat{g}_{\lambda}, using v_i = \bar{x_i}, it follows that \hat{g}_{\lambda} \in G.' l98-99: 'E_{11}' -> 'E_1' l102-103: '\pi(f_{\rho})' -> 'f_{\rho}' [2x]; l106-109: similarly [3x]. In other words, the \pi clipping is superfluous on f_\rho. l112-113: I guess a \sum_i is missing on the r.h.s of the inequality. l113: 'inequality is follows from Hoder inequality' -> 'inequality follows from the Cauchy-Schwarz inequality' ['is': superfluous, 'Hoder' -> 'Holder' -> 'Cauchy-Schwarz' (it is sufficient)] l114: the inequality in Cauchy-Schwarz holds under equality of the arguments up to multipliers: \sqrt{1-L_{ii}} / \sqrt{p_i} ||K_i||_2 k^{-1} = \sqrt{p_i}; the conclusion is OK.