Learning with Group Invariant Features: A Kernel Perspective
We analyze in this paper a random feature map based on a theory of invariance (I-theory) introduced in [1]. More specifically, a group invariant signal signature is obtained through cumulative distributions of group-transformed random projections. Our analysis bridges invariant feature learning with kernel methods, as we show that this feature map defines an expected Haar-integration kernel that is invariant to the specified group action. We show how this non-linear random feature map approximates this group invariant kernel uniformly on a set of N points. Moreover, we show that it defines a function space that is dense in the equivalent Invariant Reproducing Kernel Hilbert Space. Finally, we quantify error rates of the convergence of the empirical risk minimization, as well as the reduction in the sample complexity of a learning algorithm using such an invariant representation for signal classification, in a classical supervised learning setting.
Haar-group invariant kernels have the form:K(x,z) = \int_G \int_G k(gx, g'z) dm(g) dm(g'), (1)where G is a compact unitary group. Therefore, we say these kernels are group invariant since k(x,z) = k(gx,g'z) for any pair of transformations (g,g') belonging to the group G. For example, if we deal with images, an useful group G could a collection of translations and rotations.Since the double integral in (1) is expensive to compute, the authors propose to use sampling to compute a finite dimensional, randomized feature map that approximates the kernel K. In particular, the feature map is constructed by sampling random elements g from G using the measure m(g), sampling "templates" t from a reference distribution (which is set to be Gaussian in this paper), and then constructing the discretized CDF of the random variable $\langle gt,x \rangle$, for an input x.The construction is accompanied with concentration inequalities that depict how fast does this randomized feature map converge to the true Haar-group-invariant kernels, and an adaptation of the risk bounds from [Rahimi and Recht, 2007] for the introduced group-invariant feature map.Question: how does the number of templates affect the quality of approximation? It seems like all asymptotic guarantees are wrt n (the number of bins in the CDF discretization) and not wrt m (the number of templates, as played with in the experiments). What is the learning rate between K_s and K?The numerical simulations seem preliminary, as the method is demonstrated only on two small-scale datasets. Furthermore, there is no comparison to other related methods (such as regular random features where the number of them is increased at no cost thanks to, for example, a Fastfood approximation). Also, the plots are difficult to interpret: perhaps a logarithmic scale on the Y-axis would help.Question: how would you learn G if you are not sure about the nature of your data? This approach seems to invoke the human to be more in the loop, drifting one step away from automatic machine learning.The technical quality of the paper is good, and it is fairly well written. The significance of the paper is undermined by the minimal experimental section, which is conducted on small-scale digit datasets (this Section would be greatly strengthened by incorporating competing methods such as large random feature ensembles, histogram random features, convolutional neural networks... And extending the simulations to larger datasets such as natural images ).