Published as a conference paper at ICLR 2018 ACTIVE LEARNING FOR CONVOLUTIONAL NEURAL NETWORKS: A CORE-SET APPROACH
Convolutional neural networks (CNNs) have been successfully applied to many recognition and learning tasks using a universal recipe; training a deep model on a very large dataset of supervised examples. However, this approach is rather restrictive in practice since collecting a large set of labeled images is very expensive. One way to ease this problem is coming up with smart ways for choosing images to be labelled from a very large collection (i.e. active learning). Our empirical study suggests that many of the active learning heuristics in the literature are not effective when applied to CNNs in batch setting. Inspired by these limitations, we define the problem of active learning as core-set selection, i.e. choosing set of points such that a model learned over the selected subset is competitive for the remaining data points. We further present a theoretical result characterizing the performance of any selected subset using the geometry of the datapoints. As an active learning algorithm, we choose the subset which is expected to yield best result according to our characterization. Our experiments show that the proposed method significantly outperforms existing approaches in image classification experiments by a large margin.
After reading rebuttals from the authors: The authors have addressed all of my concerns. THe additional experiments are a good addition.
************************
The authors provide an algorithm-agnostic active learning algorithm for multi-class classification. The core technique is to construct a coreset of points whose labels inform the labels of other points.  The coreset construction requires one to construct a set of  points which can cover the entire dataset. While this is NP-hard problem in general, the greedy algorithm is 2-approximate. The authors use a variant of the greedy algorithm along with bisection search to solve a series of feasibility problems to obtain a good cover of the dataset each time.  This cover tells us which points are to be queried. The reason why choosing the cover is a good idea is because under suitable Lipschitz continuity assumption the generalization error can be controlled via an appropriate value of the covering radius in the data space.  The authors use the coreset construction with a CNN to demonstrate an active learning algorithm for multi-class classification. 
The experimental results are convincing enough to show that it outperforms other active learning algorithms. However, I have a few major and minor comments.
Major comments:
1. The proof of Lemma 1 is incomplete. We need the Lipschitz constant of the loss function. The loss function is a function of the CNN function and the true label. The proof of lemma 1 only establishes the Lipschitz constant of the CNN function. Some more extra work is needed to derive the lipschitz constant of the loss function from the CNN function. 
2. The statement of Prop 1 seems a bit confusing to me. the hypothsis says that the loss on the coreset = 0. But the equation in proposition 1 also includes the loss on the coreset. Why is this term included. Is this term not equal to 0?
3. Some important works are missing.  Especially works related to pool based active learning, and landmark results on labell complexity of agnostic active learning.
UPAL: Unbiased Pool based active learning by Ganti & Gray. http://proceedings.mlr.press/v22/ganti12/ganti12.pdf
Efficient active learning of half-spaces by Gonen et al. http://www.jmlr.org/papers/volume14/gonen13a/gonen13a.pdf
A bound on the label complexity of agnostic active learning. http://www.machinelearning.org/proceedings/icml2007/papers/375.pdf
4.  The authors use L_2 loss as their objective function. This is a bit of a weird choice given that they are dealing with multi-class classification and the output layer is a sigmoid layer, making it a natural fit to work with something like a cross-entropy loss function. I guess the theoretical results do not extend to cross-entropy loss, but the authors do not mention these points anywhere in the paper. For example, the ladder network, which is one of the networks used by the authors is a network that uses cross-entropy for training.
Minor-comment: 
1. The feasibility program in (6) is an MILP. However, the way it is written it does not look like an MILP. It would have been great had the authors mentioned that u_j \in {0,1}. 
2. The authors write on page 4, "Moreover, zero training error can be enforced by converting average loss into maximal loss". It is not clear to me what the authors mean here. For example, can I replace the average error in proposition 1, by maximal loss? Why can I do that? Why would that result in zero training error?
On the whole this is interesting work and the results are very nice. But, the proof for Lemma 1 seems incomplete to me, and some choices (such as choice of loss function) are unjustified. Also, important references in active learning literature are missing.