On Optimal Generalizability in Parametric Learning
We consider the parametric learning problem, where the objective of the learner is determined by a parametric loss function. Employing empirical risk minimization with possibly regularization, the inferred parameter vector will be biased toward the training samples. Such bias is measured by the cross validation procedure in practice where the data set is partitioned into a training set used for training and a validation set, which is not used in training and is left to measure the outof-sample performance. A classical cross validation strategy is the leave-one-out cross validation (LOOCV) where one sample is left out for validation and training is done on the rest of the samples that are presented to the learner, and this process is repeated on all of the samples. LOOCV is rarely used in practice due to the high computational complexity. In this paper, we first develop a computationally efficient approximate LOOCV (ALOOCV) and provide theoretical guarantees for its performance. Then we use ALOOCV to provide an optimization algorithm for finding the regularizer in the empirical risk minimization framework. In our numerical experiments, we illustrate the accuracy and efficiency of ALOOCV as well as our proposed framework for the optimization of the regularizer.
This paper proposes an efficiently computable approximation of leave-one-out cross validation for parametric learning problems, as well as an algorithm for jointly learning the regularization parameters and model parameters. These techniques seem novel and widely applicable.
The paper starts out clearly written, though maybe some less space could have been spent on laying the groundwork, leaving more room for the later sections where the notation is quite dense.
Can you say anything about the comparsion between ALOOCV and LOOCV evaluated on only a subset of the data points (as you mention in l137-140), both in terms of computation cost and approximation accuracy?
Other comments:
l75: are you referring to PRESS? Please name it then.
l90: "no assumptions on the distribution" -- does that mean, no prior distribution?
Definition 7: the displayed equation seems to belong to the "such that" in the preceding sentence; please pull it into the same sentence. Also, I find it odd that an analytic function doesn't satisfy this definition (due to the "there exists one and only one"). What about a two-dimensional function that has non-differentiabilities in its uppper-right quadrant, so that along some cross sections, it is analytic?
l186-187: (relating to the remarks about Def7) This sounds a bit odd; it might be better to say something like "We remark that the theory could be extended to ...".
l250: Are you saying that you are not jointly learning the regularization parameter in this second example? If the material in section 4 doesn't apply here, I missed why; please clarify in that section.
Typographic:
l200: few -> a few
References: ensure capitalization of e.g. Bayes by using {}