The Sample Complexity of Semi-Supervised Learning with Nonparametric Mixture Models
We study the sample complexity of semi-supervised learning (SSL) and introduce new assumptions based on the mismatch between a mixture model learned from unlabeled data and the true mixture model induced by the (unknown) class conditional distributions. Under these assumptions, we establish an Ω(K log K) labeled sample complexity bound without imposing parametric assumptions, where K is the number of classes. Our results suggest that even in nonparametric settings it is possible to learn a near-optimal classifier using only a few labeled samples. Unlike previous theoretical work which focuses on binary classification, we consider general multiclass classification (K > 2), which requires solving a difficult permutation learning problem. This permutation defines a classifier whose classification error is controlled by the Wasserstein distance between mixing measures, and we provide finite-sample results characterizing the behaviour of the excess risk of this classifier. Finally, we describe three algorithms for computing these estimators based on a connection to bipartite graph matching, and perform experiments to illustrate the superiority of the MLE over the majority vote estimator.
# Summary The authors try to derive the sample complexity of semi-supervised learning with mixture models, when the unsupervised mixture model may not (exactly) match the mixture model implied by the class conditional distributions of the underlying true distribution. This leads to a much more general analysis than earlier work in SSL, both considering misspecification of the mixture and more than 2 classes. They propose several methods to recover the true mapping of decision regions to classes, for which they show both the sample complexity and show empirical results of the probability of correct recovery in three example simulations.  # Main comments I would like to thank the authors for this interesting work that further advances our understanding of sample complexities when using mixture model in a semi-supervised setting. While I like the analysis presented in the work, I have some questions, concerns and suggestions whose resolution might help me appreciate the work more.  First off, I would suggest reflecting the focus on mixture models in the title, to make it clearer what to expect.  Secondly, I have some confusion over the quantify of interest in the analysis. The goal is recovering pi^*. This is perfectly clear in the Lambda=Lambda^* setting, as was also considered in earlier work. However, it is never made precise in the paper (or I missed it) what pi^* represents if Lambda!=Lambda^*. Is it simply the best possible mapping for that particular Lambda (so like (4) but for the expected log likelihood)? Aside from this, how crucial is the restriction that the mixture contains the same number of components as the number of classes? Would it not be enough for the number of components to be bigger than K, or is this not a problem because the components themselves can be arbitrarily complex?  Third, I wonder if you could comment on the following. It seems that up except for section 4.4, the analysis does not really depend on being in a semi-supervised setting: regardless of where the mixture model comes from, your results that you can find a mapping from components to classes efficiently continues to hold. In this section, however, the rate in terms of the number of unlabeled samples is rather vague (but still more insightful than papers that only consider clairvoyance). Is there more that can be said about this rate. And for reasonable discrepancies between Lambda and Lambda^* (what are reasonable values for W_1), is it reasonable to assume that the bound in Theorem 4.7 becomes informative?  Fourth, I have three questions regarding the experiments. 1. What do the curves look like in terms of classification performance (perhaps these could be added to the appendix, including a supervised baseline). 2. Some of the curves seem to flatten out even though the bounds would suggest that they correct mapping should be obtained with high probability with some number n of samples. Could you explain the relationship between the results in the bounds and the curves (is n simply very large in these cases?).  Finally, the paper has a rather abrupt ending. I understand there are space constraints, but it would be nice if the paper could end with something like a concluding sentence.  In summary, I find the pros of the paper that it: offers an interesting extension to earlier analyses of mixture models in ssl; is clearly written; has a proper analysis and explanation of the differences between the proposed approaches. I find the cons that: as in earlier work, the focus is mostly on the labeled part of ssl, while the unlabeled part (section 4.4) is less extensive. I am looking forward to the author's response to my questions to better inform my opinion of the paper.  # Minor suggestions 15: MLE is not yet defined here 18: "more and more expensive": I'm not sure labeling data has gotten more expensive, labeling all the data might have gotten more expensive because there is more of it. 85: from this definition of M_K it is unclear to me why its elements also contain the lambda_k's, instead of just the f_k's. 101: supervised -> semi-supervised would also make sense here 138: "which is a more difficult problem": in what way? 194: the simplest 231: not -> note Figure 2: Perhaps the legibility of the figure can be improved by making the figure slightly bigger, for instance by sharing the axis annotations of the small multiples.   # After response Thank you for your response. Perhaps I had not made clear enough what my confusion with pi^* was in my review, since your response does not specifically address my confusion (or I simply don't understand it): While I understand pi^* is an interesting mapping to estimate if we have Lambda^*, if we have some \hat{Lambda} from the unlabeled data, what is the mapping we are trying to estimate (not necessarily pi^* I suppose), and how does it relate to pi^*. Hopefully this clarifies my question. I would also like to reiterate my suggestion to include "mixture models" or something related in the title to make the goal of the paper clearer up front. Overall, I still find it an interesting paper but I am not more convinced of its merits than I already was before the response.