Learning Kernels Using Local Rademacher Complexity
We use the notion of local Rademacher complexity to design new algorithms for learning kernels. Our algorithms thereby benefit from the sharper learning bounds based on that notion which, under certain general conditions, guarantee a faster convergence rate. We devise two new learning kernel algorithms: one based on a convex optimization problem for which we give an efficient solution using existing learning kernel techniques, and another one that can be formulated as a DC-programming problem for which we describe a solution in detail. We also report the results of experiments with both algorithms in both binary and multi-class classification tasks.
Note: References present in the paper are referred to 
      by their numerical citations as used in the paper. Additional references 
      are included as Natbib style citations. The complete citations are given 
      at the end of the review. I have read the author rebuttal 
      carefully and I would like to retain my original review and assessment. 
      Summary ======================================== The paper 
      addresses the problem of multiple kernel learning wherein a kernel K_\mu 
      must be learned that is a positive combination of a set of M base kernels 
      K_mu = \sum \mu_i K_i. The aim of this paper is to formulate the kernel 
      learning problem in a way that yields tight learning theoretic guarantees. 
      The motivation for the approach comes from some existing learning 
      theoretic analyses for the kernel learning problem [12,18] that 
      demonstrate that "fast" O(1/n)-style convergence guarantees can be 
      achieved by analyzing the local Rademacher averages of the hypothesis 
      class. These analyses indicate that it is the "tail sum" of the 
      eigenvalues (sum of all but a few top eigenvalues) rather than the 
      complete trace that decides the convergence properties. This leads 
      the authors to formulate the kernel learning problem with "tail-sum" 
      regularization instead of trace-norm regularization (the cut-off point for 
      the tail being a tunable parameter). However, the resulting class H1 turns 
      out to be concave which prompts the authors to propose two learning 
      formulations 1) Algorithm 1: A convexification of the class H1 is 
      proposed that yields a class H2 that instead regularizes using a lower 
      bound on the tail sum of eigenvalues of the combined kernel. For this 
      class, existing L1-norm MKL solvers are applicable and are used. 
      2) Algorithm 2: To work with the concave class H2 itself, the 
      authors propose a DC programming approach wherein at each stage, the 
      concave constraint is linearized and the resultant (convex) problem is 
      solved via an alternating approach. Some interesting properties of 
      the two classes H1 and H2 are analyzed as well. For instance, it is shown 
      that H2 is not simply the convex closure of H1. Generalization guarantees 
      are given for both learning formulations with the excess risk behaving as 
      O(1/n) for base kernels with spectra that die down quickly (e.g. the 
      Gaussian kernel). The approach, being motivated from learning theoretic 
      considerations, has good generalization properties (which albeit follow 
      rather painlessly from existing results). Experiments are 
      conducted with standard L1, L2 as benchmarks. The trivial uniform 
      combination of kernels is also taken as a benchmark. Performance 
      improvements over these three benchmarks are reported on the transcription 
      start site detection problem where both Algorithm 1 and 2 give moderate 
      performance improvements (being the best on all training set sizes 
      considered in terms of area under the ROC curve). Both Algorithm 1 and 2 
      themselves give similar performance. It is also shown that the proposed 
      approach can exploit moderately informative kernels which the L1 approach 
      can discard. Experiments are also carried out on some multi-class 
      datasets where performance improvements are reported for Algorithm 1 over 
      the uniform kernel combination and standard L1 and L2 MKL formulations. 
      Questions ---------------------------------------- 1) 
      Theoretical results are given for the squared loss but the experiments use 
      the standard hinge loss. Can this gap be bridged for a more complete 
      presentation (in a future manuscript perhaps)? Hinge loss might be unable 
      to guarantee fast rates unless low noise assumptions are made but still, 
      any such analysis would give a more complete picture. Authors may also try 
      kernelized ridge regression in experiments to see if it compares with 
      hinge loss. 2) How is the cut-off parameter \theta chosen ? The 
      experiments section mentions a model parameter h: is this treated as theta 
      ? 3) How is the proposed approach different from one that simply 
      requires the learned kernel to be low rank ? The tail-sum is minimized 
      when it is zero in which case one is looking at a low rank kernel. Since 
      minimizing rank is hard (even NP-hard in some cases), what the paper seems 
      to be proposing is some relaxed version of rank (not trace norm though) 
      that still encourages low rank). 4) The results of [8] were 
      further improved in (Hussain and Shawe-Taylor, 2011). The authors may wish 
      to reference this result. 5) It seems that the authors have 
      modified the default inter-line spacing values (possibly by modifying the 
      "baselinestretch" parameter in LaTeX). Since this is a notation heavy 
      paper, this modification is making the paper look a bit crammed. Please 
      revert to default line spacing as prescribed in the NIPS style file if 
      modifications have indeed been made. 6) Line 208: ... two distinct 
      learning kernel algorithm*s, * each based ... 7) Table 1 caption: 
      ... either l1-norm or *l2*-norm regularization. 8) References are 
      not in order in which they appear in the manuscript. Please use the unsrt 
      bibliography style to implement this. 9) DC programming can be 
      expensive: can some data be given so as to give an idea about the 
      computational costs involved in the method ? Quality 
      ======================================== This is a good paper with 
      proper theoretical analysis as well as experimental work. Clarity 
      ======================================== The paper is well written 
      with expository appendices in the supplementary material. 
      Originality ======================================== The 
      idea of using tail sum regularization is new and throws up several 
      questions since it results in a concave problem. On the algorithmic front, 
      existing results are sufficient to implement the ideas. 
      Significance ======================================== The 
      paper proposes a new technique for designing multiple kernel learning 
      algorithms that tries to formulate the learning problem in a way that will 
      lead to good learning theoretic guarantees. This could have other 
      potential applications. References 
      ======================================== Zakria Hussain, John 
      Shawe-Taylor: Improved Loss Bounds For Multiple Kernel Learning, AISTATS, 
      JMLR W&CP, (15), pages 370-377, 2011.