Learning with Invariance via Linear Functionals on Reproducing Kernel Hilbert Space
Incorporating invariance information is important for many learning problems. To exploit invariances, most existing methods resort to approximations that either lead to expensive optimization problems such as semi-definite programming, or rely on separation oracles to retain tractability. Some methods further limit the space of functions and settle for non-convex models. In this paper, we propose a framework for learning in reproducing kernel Hilbert spaces (RKHS) using local invariances that explicitly characterize the behavior of the target function around data instances. These invariances are compactly encoded as linear functionals whose value are penalized by some loss function. Based on a representer theorem that we establish, our formulation can be efficiently optimized via a convex program. For the representer theorem to hold, the linear functionals are required to be bounded in the RKHS, and we show that this is true for a variety of commonly used RKHS and invariances. Experiments on learning with unlabeled data and transform invariances show that the proposed method yields better or similar results compared with the state of the art.
This paper deals with taking into account local 
      invariances in RKHS (or GP) based classification. The approach is based on 
      integrating the invariances as extra observations in the model instead of 
      designing kernels encoding the invariance properties. The method is 
      illustrated on both simulated and real world data. The proposed 
      method is well motivated and the paper is enjoyable to read. Although 
      incorporating informations such as derivatives in the model is not new, 
      the presentation given by the authors offers a coherent framework that 
      also covers less trivial cases such as translation, rotation or scaling 
      invariances. Furthermore, the various examples of the benchmark convinced 
      me of the usefulness of the method. From my point of view the 
      overall method proposed by the authors is original and could benefit the 
      NIPS community. Minor remarks: - One term is missing in the 
      right hand side of Eq. 6 - One word is missing on page 8, line 
      424