On statistical learning via the lens of compression
This work continues the study of the relationship between sample compression schemes and statistical learning, which has been mostly investigated within the framework of binary classification. The central theme of this work is establishing equivalences between learnability and compressibility, and utilizing these equivalences in the study of statistical learning theory. We begin with the setting of multiclass categorization (zero/one loss). We prove that in this case learnability is equivalent to compression of logarithmic sample size, and that uniform convergence implies compression of constant size. We then consider Vapnik's general learning setting: we show that in order to extend the compressibility-learnability equivalence to this case, it is necessary to consider an approximate variant of compression. Finally, we provide some applications of the compressibility-learnability equivalences.
This paper studies learning with general label spaces, under either the 0-1 loss or in some cases more general losses, and establishes relationships between several quantities of interest in learning theory: namely, realizable-case PAC sample complexity, agnostic PAC sample complexity, the size of a sample compression scheme (in both realizable-case and agnostic-case variants), and the size of an approximate sample compression scheme (again, in both variants). Under the 0-1 loss, they show that the leading complexity factor in all of these quantities can be bounded by the others up to certain logarithmic factors.  Under general losses, some of these relations break down, but they still find a relation between realizable-case sample complexity and realizable-case approximate sample compression, and between agnostic-case sample complexity and agnostic-case approximate sample compression.  The paper also includes a very nice separation result, arguing that for losses other than 0/1 loss (in fact, even for 3-valued losses), realizable-case learnability does not imply agnostic learnability. Specifically, they introduce an interesting construction of a label space and loss function, such that realizable-case learning is trivial, but there are distributions in the agnostic case for which there exists a hypothesis of risk 1/2, but it is not possible to identify a classifier of risk less than 3/4 from a finite data set. They establish the negative result by combining their equivalence of approximate compressability and learnability with some combinatorial arguments on hypergraphs (of size a very very large infinity); they also sketch a much simpler construction not requiring such large infinities.  Given the setup, it seems a direct proof of the negative result (not going through the compression equivalence) is also possible, but making use of the equivalence to approximate compressability offers a nice simplification and ties the result in with the theme of the rest of the paper.
Most of the results established in the paper would, in the special case of binary classification, trivially follow from the known upper and lower bounds on sample complexity based on the VC dimension. However, the results were not previously known for multiclass learning, and other general loss functions.  The results for the 0-1 loss are not particularly surprising, but it is good to know that, for instance, in multiclass classification with the 0-1 loss, the complexity measure in the agnostic sample complexity is the same as that in the realizable-case (up to log factors, but no extra factors such as log(|Y|) not present in the realizable-case sample complexity).  They also prove a tighter lower bound than previously known for the sample complexity of uniform convergence for multiclass classification in Theorem 3.6.  The techniques used in the proofs are mostly straightforward or have appeared in other related contexts previously.  They seem primarily to be pointing out ways to combine known results and argument to reach new conclusions.  The separation result in section 4.3 may be the exception, as I do not believe I have seen this kind of construction before this paper.  There are also some novel aspects in the proof of the lower bound in Theorem 3.6.  Even if most of the arguments are  straightforward, the results add up to a fairly complete picture of  how these quantities are related in general.  Furthermore, as  demonstrated in the separation result in section 4.3, the results may be useful for simplifying proofs of learnability / non-learnability, by allowing one to focus on existence / non-existence of finite approximate compression schemes.  The paper is generally well-written and easy to read.  I did notice several grammatical mistakes, so a careful read-through to correct these would improve readability.  minor comments:  I think the upper bound in Theorem 3.6 can be improved, removing the log(1/epsilon) factor, simply by using the tail bound of Talagrand (based on the chaining technique) in place of the bound of Vapnik & Chervonenkis; this is the argument that appeared in the proof of Theorem 5 of Daniely, Sabato, Ben-David & Shalev-Shwartz (2015).  In the second proof of Theorem 4.4 (last paragraph of the full paper), the fonts for X and Y need to be made consistent.  Also, the range of the hypothesis class H should have Y_i in the union, rather than X_i. Finally, since most readers will find this second proof much more accessible than the first, it seems important to include enough details so that they can easily follow the argument.  In particular, instead of merely claiming "one can verify" points (i) and (ii), it would be best to include these arguments explicitly.