Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for many common machine learning algorithms, including logistic regression, elastic-net regression, and support vector machines. Leveraging this property of differentiability, we propose a cross-validation gradient method (CVGM) for hyperparameter optimization. Our method enables efficient optimization in high-dimensional hyperparameter spaces of the cross-validation risk, the best surrogate of the true generalization ability of our learning algorithm.
The paper considers the problem of automatic tuning of hyperparameters in machine learning models. To address this problem the authors propose to use the so called cross-validation gradients, which optimize a validation objective with respect to the hyperparameters of a model. This approach and the investigated setting falls into a class of optimization problems known as bilevel optimization. The main characteristic of this class of optimization problems is the nested structure, with an outer and inner optimization objectives/problems. The outer problem corresponds to the validation objective and it is defined via an optimal solution to the inner problem which corresponds to a training objective. The paper, however, fails to make a reference to a rather rich literature on bilevel optimization (e.g., see [3-5] and references therein).
The approach, presented as Algorithm 1, does not seem different from [1] and [2] where hyperparameter optimization was considered for (kernel) support vector machines and (kernel) ridge regression. The data is initially split into k-folds (not necessarily of identical size) and each fold is used exactly once to define a validation objective whereas the complementary folds act as training data. The validation gradient is obtained by averaging the gradients of the k validation folds. Essentially, the same algorithm with k-fold cross-validation was considered in [1]. Thus, for me there does not seem to be any novelty in this approach and the paper itself.
The experiments involve synthetic regression and classification datasets but there are no novel insights that advance what is already known about the hyperparameter optimization (e.g., see [3]). For example, there is no intuition on the geometry of the optimization problem and the optimality of the outer optimization problem which is non-convex (e.g., see [7]), or dependence of the outer solution on the accuracy of the inner solution.
References:
[1] S. Keerthi, V. Sindhwani, and O. Chapelle. An Efficient Method for Gradient-Based Adaptation of Hyperparameters. NIPS 2007.
[2] O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing Multiple Parameters for Support Vector Machines. Machine Learning, 2002.
[-3] L. Franceschi, P. Frasconi, S. Salzo, R. Grazzi, and M. Pontil. Bilevel Programming for Hyperparameter Optimization and Meta-Learning. ICML 2018.
[-4] G. Kunapuli, K.P. Bennet, J. Hu, and J-S. Pang. Bilevel Model Selection for SVMs. American Mathematical Society, 2008.
[-5] E.S.H. Neto and A.R. de Pierro. On Perturbed Steepest Descent Methods with Inexact Line Search for Bilevel Convex Optimization. Journal of Mathematical Programming and Operations Research, 2011.
[-6] B. Colson, P. Marcotte, and G. Savard. A Trust-Region Method for Nonlinear Bilevel Programming: Algorithms and Computational Experience. Computational Optimization and Applications, 2005.
[-7] M. Janzamin, H. Sedghi, and A. Anandkumar. Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Method. arXiv preprint arXiv:1506.08473v3, 2016.