Learning convex polytopes with margin
We present an improved algorithm for properly learning convex polytopes in the realizable PAC setting from data with a margin. Our learning algorithm constructs a consistent polytope as an intersection of about t log t halfspaces with margins in time polynomial in t (where t is the number of halfspaces forming an optimal polytope). We also identify distinct generalizations of the notion of margin from hyperplanes to polytopes and investigate how they relate geometrically; this result may be of interest beyond the learning setting.
The paper considers the problem of PAC learning of fat convex polytopes in the realizable case. This hypothesis class is given by intersections of t fat hyperplanes, i.e., hyperplanes with margin gamma.   Using standard results, the authors derive that the VC dimension of this class is quadratic in the inverse margin size and thus that the sample complexity is polylogerithmic in this quantity. As their main result, they provide two algorithms for finding with high probability a consistent fat polytope: one with exponential runtime in t and one polynomial greedy algorithm that, however, only guarantees to find a (t log n)-polytope. Complementary, the paper states two hardness of approximation results: one for finding an approximately consistent fat hyperplane, i.e., one with the minimum number of negative points on wrong side (and all positive correctly classified), and one for finding a consistent fat polytope with the minimum number of hyperplanes. Finally, the authors also show how their results relate to the alternative class of polytopes that separate points outside of their gamma-envelope (area with Euclidean distance less or equal to gamma from the polytope boundary).  The topic is without a doubt highly relevant to NIPS and the results potentially of high impact. However, I do have some concerns regarding how the given results fit together to back the main claim. Ultimately this may be mostly an issue of the presentation as much could come down to a more conservative description of the main results including a more explicit statement of the learning problem (in particular the role of the parameter t).  - The claimed main contribution is an algorithm for learning t-polytopes in time polynomial in t. However, the polynomial time algorithm does only guarantee to find a polytope with potentially many more sides (O(t log t) for constant epsilon and delta). In contrast, the other algorithm, which indeed finds a consistent t-polytope, is exponential in t. To what degree is it appropriate to say that either algorithm efficiently (in terms of t) learns t-polytopes?  - The hardness of approximation result of Lemma 6 appears to relate to the relevant problem of finding an approximately consistent hyperplane (however, it is insufficient to conclude hardness for t larger than 1, for some of which the problem could turn out to be easier). In contrast, Theorem 7 is dealing with the problem of approximating a consistent polytope with the minimum number of sides. How does this relate to the realizable PAC learning problem? Here, we are given a dataset and know that there is a consistent t-polytope. In any case, the proof of Theorem 7 is unclear / too brief and in particular it seems Lemma 6 is not actually used in it (it might be intended as independent complementary result; the question remains, how either relates to the problem of finding a consistent t-polytope knowing that such a polytope exists).  In any case, the final two pages of the paper with the main contributions are very dense and hard to follow. In contrast, the non-essential discussion of margins versus envelopes takes up a lot of space. I believe the extension to envelops would have been fine as a brief afterthought (with proof potentially delegated to supplementary material). Instead, the actual main results should be presented in a more detailed and didactically pleasing form. So I strongly recommend to shift paper real estate accordingly.  Some additional minor comments:  - In Lemma 2 we probably need min instead of max since we combine two valid upper bounds to the VC dimension - In the proof correct "hyperplances" - Figures 1 and 2 are very similar and probably only one is needed given the tight space limitation (see comments above) - Probably, it would be more straightforward to define a linear embedding f to be a JL transform if it satisfies the reconstruction properties (with high probability) instead of talking about "linear embeddings that satisfy the guarantees of _the_ JL transform".  - In the proof of Observation 1 two times "at least" should probably be "at most" - Last line of proof of Lemma 9: "contains within _it_ a delta-net" - Theorem 10: the first part of the bound of Lemma 2 is not reflected here