Gaussian Quadrature for Kernel Features
Kernel methods have recently attracted resurgent interest, showing performance competitive with deep neural networks in tasks such as speech recognition. The random Fourier features map is a technique commonly used to scale up kernel machines, but employing the randomized feature map means that O( −2 ) samples are required to achieve an approximation error of at most . We investigate some alternative schemes for constructing feature maps that are deterministic, rather than random, by approximating the kernel in the frequency domain using Gaussian quadrature. We show that deterministic feature maps can be constructed, for any γ > 0, to achieve error with O(e e γ + −1/γ ) samples as goes to 0. Our method works particularly well with sparse ANOVA kernels, which are inspired by the convolutional layer of CNNs. We validate our methods on datasets in different domains, such as MNIST and TIMIT, showing that deterministic features are faster to generate and achieve accuracy comparable to the state-of-the-art kernel methods based on random Fourier features.
In the case of shift-invariant kernels, one technique that was proposed for constructing the function z is random Fourier features [23]. This data-independent method approximates the Fourier transform integral (1) of the kernel by averaging Monte-Carlo samples, which allows for arbitrarily-good estimates of the kernel function k. Rahimi and Recht [23] proved that if the feature map has dimension D =Ω d 2 then, with constant probability, the approximation z(x), z(y) is uniformly -close to the true kernel on a bounded set. While the random Fourier features method has proven to be effective in solving practical problems, it comes with some caveats. Most importantly, the accuracy guarantees are only probabilistic and there is no way to easily compute, for a particular random sample, whether the desired accuracy is achieved.
Our aim is to understand to what extent randomness is necessary to approximate a kernel. We thus propose a fundamentally different scheme for constructing the feature map z. While still approximating the kernel's Fourier transform integral (1) with a discrete sum, we select the sample points and weights deterministically. This gets around the issue of probabilistic-only guarantees by removing the randomness from the algorithm. For small dimension, deterministic maps yield significantly lower error. As the dimension increases, some random sampling may become necessary, and our theoretical insights provide a new approach to sampling. Moreover, for a particular class of kernels called sparse ANOVA kernels (also known as convolutional kernels as they are similar to the convolutional layer in CNNs) which have shown state-of-the-art performance in speech recognition [22], deterministic maps require fewer samples than random Fourier features, both in terms of the desired error and the kernel size. We make the following contributions:
• In Section 3, we describe how to deterministically construct a feature map z for the class of subgaussian kernels (which can approximate any kernel well) that has exponentially small (in D) approximation error.
• In Section 4, for sparse ANOVA kernels, we show that our method produces good estimates using only O(d) samples, whereas random Fourier features requires O(d 3 ) samples.
• In Section 5, we validate our results experimentally. We demonstrate that, for real classification problems on MNIST and TIMIT datasets, our method combined with random sampling yields up to 3 times lower kernel approximation error. With sparse ANOVA kernels, our method slightly improves classification accuracy compared to the state-of-the-art kernel methods based on random Fourier features (which are already shown to match the performance of deep neural networks), all while speeding up the feature generation process.
Post-rebuttal comments: Thank you for the feedback. I have raised my score on overall rating. 
Summary: In my opinion, this is an interesting and promising direction for the use of deterministic quadrature rules in kernel approximation. The weakness of the paper is in the experiments: there should be more complete comparisons in computation time, and comparisons with QMC-based methods of Yang et al (ICML2014). Without this the advantage of the proposed method remains unclear.
- The limitation of the obtained results:
The authors assume that the spectrum of a kernel is sub-gaussian. This is OK, as the popular Gaussian kernels are in this class. However, another popular class of kernels such as Matern kernels are not included, since their spectrum only decay polynomially. In this sense, the results of the paper could be restrictive. 
- Eq. (3):
What is $e_l$? 
Corollaries 1, 2 and 3 and Theorem 4:
All of these results have exponential dependence on the diameter $M$ of the domain of data: a required feature size increases exponentially as $M$ grows. While this factor does not increase as a required amount of error $\varepsilon$ decreases, the dependence on $M$ affects the constant factor of the required feature size. In fact, Figure 1 shows that the performance is more quickly getting worse than standard random features. This may exhibit the weakness of the proposed approaches (or at least of the theoretical results).
- The equation in Line 170:
What is $e_i$?
- Subsampled dense grid:
This approach is what the authors used in Section 5 on experiments. However, it looks that there is no theoretical guarantee for this method. Those having theoretical guarantees seem not to be practically useful. 
- Reweighted grid quadrature:
(i) It looks that there is no theoretical guarantee with this method.
(ii) The approach reminds me of Bayesian quadrature, which essentially obtains the weights by minimizing the worst case error in the unit ball of an RKHS. I would like to look at comparison with this approach.
(iii) Would it be possible to derive a time complexity?
(iv) How do you chose the regularization parameter $\lambda$ in the case of the $\ell_1$ approach? 
- Experiments in Section 5:
(i) The authors reported the results of computation time very briefly (320 secs vs. 384 seconds for 28800 features in MNIST and "The quadrature-based features ... are about twice as fast to generate, compared to random Fourier features ..." in TIMIT). I do not they are not enough: the authors should report the results in the form of Tables, for example, varying the number of features. 
(ii) There should be comparison with the QMC-based methods of Yang et al. (ICML2014, JMLR2016). It is not clear what is the advantage of the proposed method over the QMC-based methods.
(iii) There should be explanation on the settings of the MNIST and TIMIT classification tasks: what classifiers did you use, and how did you determine the hyper-parameters of these methods? At least such explantion should be included in the appendix.