Optimal Rates for Random Fourier Features
Kernel methods represent one of the most powerful tools in machine learning to tackle problems expressed in terms of function values and derivatives due to their capability to represent and model complex relations. While these methods show good versatility, they are computationally intensive and have poor scalability to large data as they require operations on Gram matrices. In order to mitigate this serious computational limitation, recently randomized constructions have been proposed in the literature, which allow the application of fast linear algorithms. Random Fourier features (RFF) are among the most popular and widely applied constructions: they provide an easily computable, low-dimensional feature representation for shift-invariant kernels. Despite the popularity of RFFs, very little is understood theoretically about their approximation quality. In this paper, we provide a detailed finite-sample theoretical analysis about the approximation quality of RFFs by (i) establishing optimal (in terms of the RFF dimension, and growing set size) performance guarantees in uniform norm, and (ii) presenting guarantees in L r (1 ≤ r < ∞) norms. We also propose an RFF approximation to derivatives of a kernel with a theoretical study on its approximation quality.
Given the attention that has been drawn to the random Fourier features, it is glad to see a paper trying to given a detailed analysis on how much features it needs to have descent performance. It is definitely an important problem as well to obtain better understanding of the kernel methods and the efforts to make it work for very large scale data set.The paper is to provide optimal rates for the convergence of the kernel function in terms of the uniform and L^r norm. In the main results, the authors gave an improved a rate over the previous work: reducing the linear dependence of the diameter |S| to a logarithmic dependence.This is definitely an important result for certain settings. However, I am more curious about the consistency of random Fourier features in the probabilistic setting given a probability measure, because in many real world problems, data are usually not sampled from a uniform distribution.Moreover, I would also like to see some discussions about how the convergence of the kernel function will influence the performance of classification or other applications that use kernel method.To sum up, the paper does deliver what the authors promise, even though I didn't carefully go through the proof of the main results. The results are interesting in its own right, but I feel its contribution to the literature is relatively incremental.