Algorithmic Stability and Uniform Generalization
One of the central questions in statistical learning theory is to determine the conditions under which agents can learn from experience. This includes the necessary and sufficient conditions for generalization from a given finite training set to new observations. In this paper, we prove that algorithmic stability in the inference process is equivalent to uniform generalization across all parametric loss functions. We provide various interpretations of this result. For instance, a relationship is proved between stability and data processing, which reveals that algorithmic stability can be improved by post-processing the inferred hypothesis or by augmenting training examples with artificial noise prior to learning. In addition, we establish a relationship between algorithmic stability and the size of the observation space, which provides a formal justification for dimensionality reduction methods. Finally, we connect algorithmic stability to the size of the hypothesis space, which recovers the classical PAC result that the size (complexity) of the hypothesis space should be controlled in order to improve algorithmic stability and improve generalization.
SUMMARYThis paper defines a notion of algorithmic stability that is shown to be equivalent to uniform generalization. Different consequences for generalization are explored in connection to techniques used to control algorithmic stability directly, richness of the hypothesis space and techniques used to control it, and also a notion of size of the input space. CONTRIBUTIONS* A notion of algorithmic stability that corresponds to uniform generalization. * Theoretical justifications for different popular learning strategies used to improve algorithmic stability. * Strengthening of the understanding of generalization and: the input space, the hypothesis space, the learning algorithm.These contributions strengthen the understanding of generalization, and can lead to interesting future work that deepens the understanding of different techniques used to increase generalization (hold-out set validation, regularization, dimensionality reduction, etc.). Proposing a notion of stability that is equivalent to (uniform) generalization is challenging, and I think this makes the contributions of this paper strong. SUPPORTThe proofs are correct, simple and easy to understand. The claims that provide a justification for techniques that improve generalization are well-founded in the theoretical results. The proof sketches are informative as well, while maintaining the pace of the exposition. TECHNICAL QUALITYThe paper is technically sound, and the consequences of the central results we well-explored. The authors have defined stability on expectation, not with high-probability (in contrast to sample complexity bounds, which hold with high-probability). The expectation allows us to obtain low-probability bounds with Markov's inequality, but it is not clear whether high-probability bounds can be obtained without fixing a specific algorithm. ORIGINALITYThe paper seems well-inserted in the literature, but my knowledge of related work in the area of stability is shallow and I cannot make a credible assessment of originality. CLARITYThe paper is well-written, with clear explanations and clear proofs. This theoretically-oriented work is well executed and it is an accessible read (in this reviwer's opinion). Based on the appreciation of other reviewers, though, it seems that the clarity of the paper can be improved.FOR THE REBUTTALPlease comment on the issue of expectation bounds vs. high-probability bounds. Can high-probability bounds be obtained without fixing an algorithm, or are the expectation bounds alone a significant contribution?It would be good if the authors could comment on the relationship between their definition of stability and existing definitions, this has been raised by other reviewers too.DETAILED COMMENTS The definition of stability proposed is interesting; it is sensible, as it can be interpreted as "the output of the algorithm does not depend too much on any single example", but I was wondering how it relates to existing definitions (e.g., the ones where one observation in the sample is dropped, or where it is swapped for another one). On the other hand perhaps the point is not so much to compare them, in case the endeavour in previous works has been to characterize generalization with some notion of stability.Considering Section 5.3, do you think we should be caring about Shannon Entropy of the output hypothesis (when designing algorithms), rather than the size of the hypothesis space? I am asking this because of the issues with bad ERM learners in multiclass classification [1], in which case we have to do something besides "just ERM" to be successful.Example 2 made me think of hold-out set validation (where a validation set is used to increase the "stability" of the algorithm) and also of minimization using Bregman projections (minimize a convex loss, then do a Bregman projection on a set) -- it also seems similar to post-processing.I see that the concepts in Definition 8 match the losses you use to show that stability is necessary for uniform generalization. Do you think it allows you to conclude that classification (with the 0-1 loss) is a hardest problem from the point of view of stability/generalization? REFERENCES:[1] Daniely, Sabato, Ben-David & Shalev-Shwartz (2013). "Multiclass learnability and the ERM principle."FIXES[l:60] "nature of the observation" [l:127] I am not a fan of this notation, why not use conditional expectation? (Using the subscript of the expectation only to indicate the distribution of some r.v. is helpful, though.) [l:152] The symbolic convention somewhat breaks when you use $\mathcal{L}$ for the algorithm, so why not use simply $A$? [l:173] I would suggest removing the last sentence from this paragraph. [l:190-191] "variables are independent of each" [l:265-266] "H_2 contains less information" [l:317] Here and in the appendix you used $\mathcal{V}$ to denote the total variation, while you defined it with a $\mathcal{T}$. [l:357-358] "always holds:" [l:383] You inverted the arguments of the KL divergence here. [l:400] $\mathbb{P}(Z_{\mathrm{trn}} = z)$ (the parentheses need to be fixed). [l:420] I would suggest using verbs in the past here "In this paper, we showed" [l:421-422] "always vanishes at"(Appendix) [l:185] (double ellipsis) [l:228] Perhaps you can add the reference to H. Robbins (1955) and Shalev-Swartz & Ben-David's book at the end of the appendix, or expand the one to H. Robbins (1955) as a footnote.POST-REBUTTAL REMARKSThere seems to be little understanding of the connections between stability by conditioning vs. stability by observation elimination vs. stability by observation switching. It would seem that the definition arose as a tool that worked for the desired results, and the authors should be upfront about the limited understanding of the aforementioned connections.The authors suggest that a strength of their main result is its algorithm-independence. However this algorithm-independence is the case only for expectation guarantees on stability, not high-probability guarantees, which are often what one is interested in. On the other hand, it is known how to convert expectation guarantees for a base algorithm into high-probability guarantees for the same algorithm combined with a hold-out set validation scheme (cf. Shalev-Shwartz & Ben-David, "Understanding Machine Learning: From Theory to Algorithms"). Therefore, the authors should mention this limitation of their results. The contributions are nevertheless solid and interesting -- of course, not as strong as algorithm-independent high-probability guarantees, which could be pursued as future work.