A Non-generative Framework and Convex Relaxations for Unsupervised Learning
We give a novel formal theoretical framework for unsupervised learning with two distinctive characteristics. First, it does not assume any generative model and based on a worst-case performance metric. Second, it is comparative, namely performance is measured with respect to a given hypothesis class. This allows to avoid known computational hardness results and improper algorithms based on convex relaxations. We show how several families of unsupervised learning models, which were previously only analyzed under probabilistic assumptions and are otherwise provably intractable, can be efficiently learned in our framework by convex optimization.
This paper proposes a PAC-style learning framework for unsupervised learning which the authors call CONUS (Comparative Non-generative UnSupervised) and also proposes learning methodology. In contrast to PAC-learning, the CONUS framework is distribution-dependent hence suitable to analysis of unsupervised learning. To further support such analysis, it is non-generative, i.e. makes no assumption on joint distributions but instead evaluates learning by minimizing reconstruction error. As an illustrative example the authors show how Principle Component Analysis can be formally learned in their setting. They also give a general class of hypothesis classes for unsupervised learning which they call polynomial spectral decoding, and they show how they can be efficiently learned under their definitions via convex optimization. Their main contribution is a convex optimization-based methodology for improper learning of a class of hypothesis classes, which includes dictionary learning. For analysis of this they introduce an averaged variant on their original CONUS-learnable definition. Their new class of algorithms are polynomial time for a broad class of dictionary models, thus significantly advancing state of the art.
* I need to clarify: my low score on question 8 is only because the paper must be revised to fix very frequent typos and English issues (i.e., in its current form the paper's presentation is not NIPS-acceptable just because of these issues). From a technical point of view however the paper is clear and very well-written and I feel it makes an important contribution. My score on question 8 would have been a 4 if the English issues were not present. I therefore strongly vote for acceptance after thorough revision. * It is good that sufficient care is taken to provide background for non-experts (see some positive comments on this later on) but in some places at the very start the level of discussion is aimed a bit low, for example in lines 14-19. * It would be essential that the authors have the paper proofread for style and grammar issues. Currently, there are too many errors. Some examples (likely careless typos): a missing "is" in the sentence "it does not assume any generative model and based on..." in the abstract; should be "learner" or "learning algorithm"in the sentence "In PAC learning, the learning can access.."; the sentence "Another approach from the information theory literature studies with online lossless compression." does not make sense as stated but I'm not sure what the authors intended; "crucial" twice in a row around line 90; should be "if there exists" in line 96; "the function that compute" should be "... computes", "can efficiently solvable" in line 130, "matrice" should be "matrix" etc... * Another (relatively minor) presentation issue: the chosen typesetting/formatting style italicizes the words Definition and Theorem etc.. but leaves the body of these environments un-italicized. This does not seem standard in NIPS; more common and clearer in my opinion is boldface for Def/Thm and italics for body (also italics for the word Proof not boldface). Was the submitted version generated with the NIPS style file? Please verify. Also, when referring to a specific definition or theorem, I believe more common NIPS practice is to capitalize the word Definition/Theorem, e.g. "from Theorem X we see..." * Line 111, there is an errant period: "Notations: . " * "We keep a list of notations that we are using." - > change to for example "The following is a listing of notation that we use." * Line 126 "pretended to the new data points" should be changed. Also, state what \ell is here, or perhaps avoid this letter altogether since it is also used for the loss function. * The actual specification of the framework and discussion around it in Section 2 is well-done (in particular, here the authors assume just enough familiarity with standard theory and give a solid high-level explanation of why Theorem 2.1 holds based on this). Also high-level explanation for the proof of Theorem 3.1 is appreciated and convincing. The framework is very promising.  * The example and variant of framework in Section 4 on optimization encoding and efficient dictionary learning is nice. It is well-described. The authors also take care to explain its element of novelty on lines 231ff. The clearly presented proof overview which follows is greatly appreciated. It seems to correct to me, though I did not have time to read the detailed proof in the Appendix. * I am curious if the authors have considered auto encoders in the context of the CONUS framework. * Throughout the paper, the authors need to refer to a class of hypothesis class or a type of hypothesis class. I think it would be good to fix handling/wording for this at the start and consistently use it also in defining the framework in Section 2 + later discussions. As it is, one sees the wording "hypothesis class" in the framework definition, but "set of hypothesis" to refer to a type of hypothesis class on line 134. * The example and discussion in line 144ff is nice.