Spherical Random Features for Polynomial Kernels
Compact explicit feature maps provide a practical framework to scale kernel methods to large-scale learning, but deriving such maps for many types of kernels remains a challenging open problem. Among the commonly used kernels for nonlinear classification are polynomial kernels, for which low approximation error has thus far necessitated explicit feature maps of large dimensionality, especially for higher-order polynomials. Meanwhile, because polynomial kernels are unbounded, they are frequently applied to data that has been normalized to unit 2 norm. The question we address in this work is: if we know a priori that data is normalized, can we devise a more compact map? We show that a putative affirmative answer to this question based on Random Fourier Features is impossible in this setting, and introduce a new approximation paradigm, Spherical Random Fourier (SRF) features, which circumvents these issues and delivers a compact approximation to polynomial kernels for data on the unit sphere. Compared to prior work, SRF features are less rank-deficient, more compact, and achieve better kernel approximation, especially for higher-order polynomials. The resulting predictions have lower variance and typically yield better classification accuracy.
== Summary ==This paper proposes a Spherical Random Fourier (SRF) features for polynomial kernels for data on the unit sphere. Although the polynomial kernels on the unit sphere is known to be shift-invariant, the paper shows that it is not always a positive definite function which prevents one from applying Bochner's theorem and obtain random Fourier features. Based on Schoenberg's characterization of positive definite function on an interval [0,2], they propose to approximate the kernel as an indefinite sum of Gaussians. The optimal probability p(w) from which the random features are sampled is obtained by minimizing the MSE between the true kernel K(z) and its approximation. Compared to Random Maclaurin (RM) and Tensor Sketch (TS), the proposed technique seems to enjoy better approximation as well as predictive accuracy, especially for polynomial of large degree p. In many datasets, however, it seems to perform as good as TS in term of accuracy, e.g., USPS. It's very nice to also see the extensive experimental results in the supplementary.In summary, this is a nice paper, although I see some disconnection between motivation in using Schoenberg's result (Theorem 4). However, I believe Schoenberg's characterization of kernel can be applied more generally. The empirical results are encouraging.== Quality ==The paper is technically sound. The major criticism I have is on Section 3.2 and Section 4. Regarding Section 3.2, it is not clear why Schoenberg's characterization is a suitable way of approximating the K(z) on [0,2]. It looks like one can construct an approximation of K(z) directly on [0,2] using arbitrary functions. Better justification should be provided.In Section 4, the approximation error decays at a rate of O(p^{-2.5}) which is a bit obscure as the analysis is done with fixed N=1. How does the number of Gaussians, N, affect the approximation? In experiment, it is mentioned that the specific number of N had negligible effect on the result. Why is this the case? Can you incorporate N into the analysis in Section 4 so it becomes clearer?Some comments/questions: [Line 086-089] To interpret the Fourier transform as a probability distribution, should one also assume that K(0) = 1? == Clarity ==The paper is clearly written. I only have minor comments:The notation d^d in Eq. (1) is confusing. There seems to be a disconnection between the first part and the approximation of the kernel. Comments/questions: [Line 298] what does (2k - 1)!! mean mathematically?== Originality ==Previous works in large-scale kernel learning oftens employed Bochner's characterization of positive definite kernel. On the other hand, this paper took another path and used the characterization of Schoenberg which is less known in the literature. One can argue, however, that using Schoenberg's characterization may not be the best option.== Significance ==It looks like the Schoenberg's characterization (Theorem 4) can be applied more generally, not just to polynomial kernels.One drawback of this paper is the restriction to polynomial kernel although the Schoenberg's characterization can already be applied to more general class of shift-invariant kernels. It is not clear why this characterization is particularly useful for polynomial kernels. What is the main difficulty?Minor commentsIn many datasets, the performance of SRF is comparable to that of TS. What is the intuition?