We propose using active learning based techniques to further improve the stateof-the-art semi-supervised learning MixMatch algorithm. We provide a thorough empirical evaluation of several active-learning and baseline methods, which successfully demonstrate a significant improvement on the benchmark CIFAR-10, CIFAR-100, and SVHN datasets (as much as 1.5% in absolute accuracy). We also provide an empirical analysis of the cost trade-off between incrementally gathering more labeled versus unlabeled data. This analysis can be used to measure the relative value of labeled/unlabeled data at different points of the learning curve, where we find that although the incremental value of labeled data can be as much as 20x that of unlabeled, it quickly diminishes to less than 3x once more than 2,000 labeled example are observed.
Summarize the paper:
This paper proposes a method that can deal with an active-learning scenario for the recently proposed semi-supervised learning method: MixMatch.  More specifically, the proposed method considers uncertainty measures to choose samples and a diversification step to ensure diversity within the sampled batch.  For uncertainty measures, the paper considers the simple maximum confidence and the gap between two most likely classes.  Additional augmentation techniques inspired from MixMatch are used.  For diversification, a clustering method and an information density method are considered.  Furthermore, the paper proposes a cost analysis model to compare labeled and unlabeled samples.  Experiments demonstrate the behavior of the proposed method.
Pros of the paper:
- The experimental results seem to be strong and encouraging.
- The discussions on the cost of labeled and unlabeled samples seems to be an important contribution for semi-supervised active learning.
- The motivation and direction of the paper is simple and easy to follow: Take the state-of-the-art semi-supervised learning algorithm and propose an active-learning version of it.
- It is not a straightforward combination of MixMatch and active learning, and there are some specialized techniques such as “aug” used in the design of the proposed algorithm.
Cons of the paper:
- Only uncertainty based sampling methods are considered, but is this enough?  There seems to be no other papers that deal with active semi-supervised learning for a deep learning context, so it might be important to really explore the many sampling methods (e.g., from survey of Settles 2009).
- A more minor comment: The same issue goes for the semi-supervised learning side.  MixMatch is the state of the art in terms of accuracy for image domains, but it is an ensemble of several semi-supervised learning methods, and have strong assumptions, e.g., smoothness assumption, small distribution overlap, etc.  This will mean the proposed method will also have those strong assumptions and limits the method’s applicability.
- In experiments, it would be better to have figures that are usually used in active learning experiments, where the x-axis is the remaining budget and y-axis is the performance measure.
Additional comments:
Active learning methods gives labels to unlabeled samples in different epochs until the budget is used up, but it would be interesting to give the final labeled and unlabeled dataset after budget is used up as a fixed dataset, and then train the traditional passive MixMatch with this.  Then we can really compare the original MixMatch and active MixMatch.  If the proposed method still works better,  then the proposed method might be meaningful not only as an active learning method but also as a curriculum learning method.
********************
I would like to thank the authors for answering my questions and updating the paper, but would like to keep my score due to the 2nd point of the cons.  A minor comment on the second point of the author response:  The sharpening step in MixMatch can be regarded as an entropy minimization procedure, which I think is based on the assumption of low distribution overlap.