Learning with SGD and Random Features
Sketching and stochastic gradient methods are arguably the most common techniques to derive efficient large scale learning algorithms. In this paper, we investigate their application in the context of nonparametric statistical learning. More precisely, we study the estimator defined by stochastic gradient with mini batches and random features. The latter can be seen as form of nonlinear sketching and used to define approximate kernel methods. The considered estimator is not explicitly penalized/constrained and regularization is implicit. Indeed, our study highlights how different parameters, such as number of features, iterations, step-size and mini-batch size control the learning properties of the solutions. We do this by deriving optimal finite sample bounds, under standard assumptions. The obtained results are corroborated and illustrated by numerical experiments.
After Author Response:  Thanks to the authors for their various clarifications.  I have updated my score to an 8 accordingly.  I think the planned updates to the empirical section will add a lot of value. ============= Summary: This paper analyzes the generalization performance of models trained using mini-batch stochastic gradient methods with random features (eg, for kernel approximation), for regression tasks using the least squares loss. Their main theorem (Theorem 1) bounds the gap in generalization performance between the lowest risk model in the RKHS with the SGD trained model after t mini-batch updates; the bound is in terms of the learning rate, the mini-batch size, the number of random features M, the training set size n, and t.  They show that under certain choices of these parameters, M = O(sqrt(n)) features are sufficient to guarantee that the gap is <= 1/sqrt(n). This is the learning rate achieved by the exact kernel model, which has been proven to be optimal. This shows that instead of paying O(n^3) time and O(n^2) space to solve the exact kernel problem, one can spend O(dn * sqrt(n)) time and O(sqrt(n)d) space and solve the problem using random features and stochastic gradient descent.  These are significant savings.  The theorem also suggests that the learning rate should increase as the mini-batch size increases; this is validated empirically.  The paper also presents experiments showing that classification error saturates for M << n.  The work most relevant to this paper is the paper from NIPS of last year by Rudi and Rosasco: "Generalization Properties of Learning with Random Features" (I will refer to this paper as [RR17]). That paper proves a result quite similar to the one in this paper, but specifically for the kernel ridge regression estimator learned with random features (closed form solution, Equation 7 in that paper). The present work extends that result to the model learned using mini-batch SGD with random features. This is significant because in the large scale setting SGD is typically the method of choice. Solving for the KRR estimator requires storing the full random features covariance matrix in memory in order to invert it; this could be prohibitive when the number of random features is very large. Stochastic methods are thus better suited for the large-scale setting.  I will now discuss the quality, clarity, originality, and significance of this work: 1) Quality: Quality of the work is high. The paper solves a challenging theoretical question using sophisticated analysis.  The experiments are a weaker point of the paper.  Figure 1b is ok, showing how larger mini-batches should be paired with larger learning rates; though perhaps each column should be converted into a line on a plot. Figure 1a in my opinion could be presented a lot more clearly; I would prefer seeing plots on various datasets showing how performance varies as you increase the number of random features (after training has finished, for example). Showing results using color-scale for all 10 passes of the data is not a good use of space in my opinion.  2) Clarity: The work is presented quite clearly. For example, I appreciated how the corollaries explained how the various parameters could be set to achieve the desired rates.  I also liked how in lines 155-162 describes clearly what happens as you increase the batch size/learning rate.  3) Originality: This work is relatively original.  It extends recent results [RR17] to a new training setting.  I'm not certain how novel the proof techniques are.  4) Significance: I believe this work is an important step toward understanding the performance of models learned using random features with mini-batch SGD in general (the results in this paper only apply to least squares loss).  It appears that in practice learning models using SGD and random features is very effective; this work helps explain why.  This work also provides a way of understand how different SGD hyperparameters should be chosen together in order to achieve small excess risk --- this insight is relevant in practice.  Questions: -- Couldn't a result similar to the one in this paper follow pretty directly from the [RR17] result?  In particular, SGD on the regularized least squares objective should converge to the optimum at a certain rate; the optimum of this objective is precisely given by the closed form ridge regression solution, for which the [RR17] results presents generalization bounds. -- Could these results be extended to fastfood features? That would reduce the time to O(log(d) n * sqrt(n)) and the space to O(sqrt(n)).  NIT:  - Section 2.1:  'D' is used to denote dimension of data space X.  In section 2 (lines 68-73) 'd' is used. - Section 3.2: 'L' is never defined.  And in the equation for Cf(x), there should be an "f" before the "(x')\rho_X(x')". - Lines 188-189: "The smaller is \alpha the more stringent...".  I believe it should be: "The larger \alpha is, the more stringent..." - I don't think the acronym SGM is ever defined (stochastic gradient methods?) - Line 244: SMG should be SGM. - Lines 35-36 should cite [RR17]  [RR17] Alessandro Rudi, Lorenzo Rosasco: Generalization Properties of Learning with Random Features. NIPS 2017