Scalable Kernel Methods via Doubly Stochastic Gradients
The general perception is that kernel methods are not scalable, so neural nets become the choice for large-scale nonlinear learning problems. Have we tried hard enough for kernel methods? In this paper, we propose an approach that scales up kernel methods using a novel concept called "doubly stochastic functional gradients". Based on the fact that many kernel methods can be expressed as convex optimization problems, our approach solves the optimization problems by making two unbiased stochastic approximations to the functional gradient-one using random training points and another using random features associated with the kernel-and performing descent steps with this noisy functional gradient. Our algorithm is simple, need no commit to a preset number of random features, and allows the flexibility of the function class to grow as we see more incoming data in the streaming setting. We demonstrate that a function learned by this procedure after t iterations converges to the optimal function in the reproducing kernel Hilbert space in rate O(1/t), and achieves a generalization bound of O(1/ √ t). Our approach can readily scale kernel methods up to the regimes which are dominated by neural nets. We show competitive performances of our approach as compared to neural nets in datasets such as 2.3 million energy materials from MolecularSpace, 8 million handwritten digits from MNIST, and 1 million photos from ImageNet using convolution features.
The paper proposes a large-scale kernel machine variant based on a combination of stochastic functional gradient descent and kernel random features. It is shown that the resulting estimate of the gradient is unbiased, thereby opening the door to convergence proofs. Generalization bounds are given, and the paper concludes with collection of experiments mainly involving datasets enlarged with virtual examples.The clarity, organization, and flow of the paper is good (although there are some grammatical errors that should be corrected).  It is clear the authors gave some thought to their presentation.  A noteworthy aspect of this submission is that it is thorough, particularly in light of the space constraints. The authors explain where the problem is, how their proposed solution addresses those problems, provide some theory to ensure the suggested algorithm is reasonable, and test their ideas on multiple, relatively realistic datasets against some existing alternatives. The theoretical component may, perhaps, be seen as among the more interesting contributions of the paper (more on why below). The observation that one need only store a random seed and a pseudo-RNG routine to reproduce a sequence of random projections is a nifty one. The experiments constitute a decent illustration of the method, and the authors should be commended for providing the additional experimental detail given in the supplementary material. A concern with the paper is a possible lack of novelty and significance. Fundamentally, the main idea is to combine two pre-existing ideas: functional gradient descent and random features approximating a (shift-invariant) kernel. Large n is the target problem addressed by both, but the key motivation for using kernel random features, and for increasing complexity, is to allow to reduce test-time storage by exchanging memory for additional computation. However, the paper should address the following  important questions around this reasoning:- It is true that testing requires minimal storage, under the assumption that you only really need to store the RNG seed from which one can then re-generate the random projections assumed during training. But this is also true for the vanilla random features approach anyways. - The proposed method exchanges low test-time memory footprint for a potentially huge, redundant computational burden associated with repeated random number generation and projection steps. The regime where this becomes acceptable could be for very large n indeed, particularly since random projections are computed anew during training iterations. It could be argued, even without random projections of any kind, that for all but the largest training sets, the test time storage burden is less of a concern. O(n) -- a single pass over the data -- may not be all that bad for many large applications. If random features are used, then a similar argument might be made for storing the projections. - The functional gradient descent approach comes with its own drawbacks. It may only be useful for more exotic loss functions, since if we use random features alone in the setting where (n is much larger than p), the square loss or hinge loss will result in a very small linear problem with random features alone. SGD may not be necessary.In light of these questions, the method may not have the strongest practical implications. The theory, if correct, could however serve as an interesting illustration of martingale methods, since the (mathematical) problem is clear, and the goals are relatively self-contained.