Generalization in Adaptive Data Analysis and Holdout Reuse *
Overfitting is the bane of data analysts, even when data are plentiful. Formal approaches to understanding this problem focus on statistical inference and generalization of individual analysis procedures. Yet the practice of data analysis is an inherently interactive and adaptive process: new analyses and hypotheses are proposed after seeing the results of previous ones, parameters are tuned on the basis of obtained results, and datasets are shared and reused. An investigation of this gap has recently been initiated by the authors in [7], where we focused on the problem of estimating expectations of adaptively chosen functions. In this paper, we give a simple and practical method for reusing a holdout (or testing) set to validate the accuracy of hypotheses produced by a learning algorithm operating on a training set. Reusing a holdout set adaptively multiple times can easily lead to overfitting to the holdout set itself. We give an algorithm that enables the validation of a large number of adaptively chosen hypotheses, while provably avoiding overfitting. We illustrate the advantages of our algorithm over the standard use of the holdout set via a simple synthetic experiment. We also formalize and address the general problem of data reuse in adaptive data analysis. We show how the differential-privacy based approach given in [7] is applicable much more broadly to adaptive data analysis. We then show that a simple approach based on description length can also be used to give guarantees of statistical validity in adaptive settings. Finally, we demonstrate that these incomparable approaches can be unified via the notion of approximate max-information that we introduce. This, in particular, allows the preservation of statistical validity guarantees even when an analyst adaptively composes algorithms which have guarantees based on either of the two approaches.
# SummaryThe paper considers the setting where an analyst has a training set and a hold-out set from the same distribution. The analyst constructs her methods based on the training set, but also repeatedly performs checks on the hold-out set to make important decisions. In general this repeated use of the hold-out set may lead to overfitting to the hold-out set, but if the analyst only performs her checks using the methods introduced in the paper, overfitting is avoided, as shown by the main results: Thms 9 and 10. This extends earlier results based on differentially private methods.# Positive PointsWhat is exciting about this line of results, is that they do not a priori restrict the class of functions (say, to a class of limited VC dimension) from which the analyst can select her checks. Otherwise, standard uniform concentration inequalities for VC classes would suffice. See lines 226-232 in the paper.An interesting point of the paper is that it goes beyond differentially private methods to also include algorithms whose output can be described by a small number of bits. This generalization is based on a unifying quantity called (approximate) max-information. Unlike notions of algorithmic stability, which are known to be closely related to generalization, this max-information allows composing multiple algorithms in a sequence. (See lines 234-243.)The high-level discussion in the paper is generally well-written. # Negative PointsIn spite of all the positive points mentioned above, I have some significant concerns about Thm 9 (one of the main results), which currently makes me believe that the paper would need a serious revision before it could be accepted.# Main concernsRegarding Thm 9: 1. The theorem makes a deterministic statement "for all i such that  a_i \neq \perp", but whether "a_i \neq \perp" is random, because it  depends on the adaptively chosen functions phi_i, on the randomness  in the data, and on the randomness introduced by the algorithm. 2. The in-probability statement bounds the probability that a_i deviates  significantly from the expectation of phi_i for a given fixed i. But  if the analyst adaptively makes decisions based on the output of the  algorithm, then a_i must be close to the expectation of phi_i for all  i simultaneously. Otherwise the analysis might take a wrong turn  somewhere. 3. I have been unable to verify the claim in lines 119,120 of the  introduction, which I paraphrase as: the number of queries m can be  exponential in n as long as the budget B is at most quadratic in n.  In particular, the result does not appear to deteriorate with m,  which I think is because of point 2 above. And in order for the  result to be useful to the analyst, I would expect that tau would go  to 0 with n, for example as tau ~ 1/sqrt(n). But if we let B be  quadratic in n, then having tau go to 0 will make beta go to  infinity, and hence the result becomes vacuous. So I don't know which  parameters to plug into the theorem. About the experiments: why do you construct a classifier that only uses weights +1 and -1 based on the signs of correlations? Is there a plausible scenario in which someone might do this in practice? # Minor remarksAs acknowledged by the authors in their proof of Thm 23 in the additional material, the extension to algorithms whose output can be described using a small number of bits can also be obtained from a union bound argument. This is hinted at in lines 171-179 of the introduction, but may not be entirely clear from the discussion there. # A potential strengtheningFinally, I would like to point out a possible strengthening of the results, which may or may not be useful.Technically, all results are based on a change of measure from the joint distribution P of a sample and an algorithm's output on that sample to the independent product distribution Q of the two (Thm 4). If P and Q are sufficiently close in terms of "max-information", then we may essentially still treat the sample as fresh even if we have already looked at the output of the algorithm.It might be of interest to observe that, at least for beta = 0, the bound on max-information can be relaxed. To do this, the max-information may be interpreted as the Renyi divergence D_alpha(P||Q) of order alpha = infinity. Renyi divergence is increasing in alpha (see e.g. [1]), so requiring a bound on alpha=infty is the strongest requirement one can impose. Thm 4 then states that we can change measures from P to Q if D_infty(P||Q) is sufficiently small. It is actually possible to change measures under the weaker requirement that D_alpha(P||Q) is small for any alpha > 1. For alpha=1 we recover regular mutual information, which, as the authors point out, is not strong enough to allow a change of measure.The proof is essentially a special case of Lemma 1 of [2], but the relation may be hard to see, so allow me to translate. Let X be the indicator random variable, which is 1 if (S,A(S)) in cal{O} and 0 otherwise. Then read Lemma 1 of [2] with E_P[X] instead of L_P(h,f), E_Q[X] instead of L_Q(h,f), and M = max X = 1 to get: E_P[X] <= ( 2^{D_alpha(P||Q)} * E_Q[X] )^{(alpha-1)/alpha} * M^(1/alpha)which is equivalent to P[(S,A(S)) in cal{O}] <= ( 2^{D_alpha(P||Q)} * Q[(S,A(S)) in cal{O}] )^{(alpha-1)/alpha}As alpha -> infinity, we recover Thm 4, but the result also holds for smaller alpha.1. Van Erven, Harremoes, "Renyi Divergence and Kullback-Leibler Divergence, IEEE Transactions on Information Theory, 2014. 2. Mansour, Mohri, Rostamizadeh, "Multiple Source Adaptation and the Renyi Divergence", UAI 2009.  # Minor issues (typo's etc)Line 114: training set -> hold-out set (I believe)Thm 4: two issues: "k = I_infty^beta(S;A(S)) = k" -> I_infty^beta(S;A(S)) <= k?Thm 8: The sentence "Let Y = ... on input S." can be removed.Lines 314-323: the bounds you mention bound the probability that the empirical estimate is more than tau *away* from the true accuracy.