This paper proposes ASAL, a new pool based active learning method that generates high entropy samples. Instead of directly annotating the synthetic samples, ASAL searches similar samples from the pool and includes them for training. Hence, the quality of new samples is high and annotations are reliable. ASAL is particularly suitable for large data sets because it achieves a better run-time complexity (sub-linear) for sample selection than traditional uncertainty sampling (linear). We present a comprehensive set of experiments on two data sets and show that ASAL outperforms similar methods and clearly exceeds the established baseline (random sampling).
In the discussion section we analyze in which situations ASAL performs best and why it is sometimes hard to outperform random sample selection. To the best of our knowledge this is the first adversarial active learning technique that is applied for multiple class problems using deep convolutional classifiers and demonstrates superior performance than random sample selection 1 .
The paper presents a pool-based active learning method that achieves sub-linear 
runtime complexity while generating high-entropy samples, as opposed to linear 
complexity of more traditional uncertainty sampling (i.e., max-entropy) methods. 
This is achieved by using a generative adversarial network (GAN) to generate 
high-entropy samples that are then used by a nearest neighbor method to pick 
samples from a pool, that are closest to the generated samples. The sub-linear 
complexity is achieved through the use of a k-d tree, combined with the fact 
that similarity is computed on the feature space and samples can thus be indexed 
once (as the feature space does not change while training).
The proposed idea builds on top of previously published work on Generative 
Adversarial Active Learning (GAAL). The main difference is the added nearest 
neighbor component, as GAAL is directly using the generated examples, thus 
achieving constant runtime complexity, rather than sub-linear.
I like the overall direction and the idea of being able to perform uncertainty 
sampling in sub-linear time. The approach is interesting. However, the results 
presented in the paper are not strong and I do not see whether or not I should 
be using this method over uncertainty sampling. Most importantly, the results 
are strongest only for the MNIST experiments, which are over a small dataset. 
Given that the method is motivated by the scalability argument, I would like to 
see at least one large scale experiment where it performs well, and more 
specifically, outperform random sampling. Also, I would really like to see a 
more principled and thorough experimental investigation with justifications for 
the configurations used and with more comparisons to alternative approaches, 
such as GAAL, which has constant complexity.
I believe that it would be better for your paper if you work a bit more on the 
experimental evaluation and submit a revised version at a later deadline.
== Background and Method ==
The background and method sections are clear and easy to follow. One improvement 
I can see is making figure 1 more clear, by maybe explicitly stating in the 
figure what "G" and "F" are. One more point of interest is that the way you 
perform sample matching makes some smoothness assumption about the feature 
space as related to the classifier uncertainty. I perceive this as a smoothness 
assumption on the decision boundary of the classifier and I do not know how true 
is may be for deep neural networks, but I can see how it may be true for 
logistic regression models and support vector machines (SVMs), depending on the 
kernel used. I believe that this point and main assumption may be worth further 
discussion, given that it is also about the main difference your method has with 
respect to GAAL.
I do not have any other major comments for these sections as my main concerns 
are about the experiments section.
== Experiments ==
In the experiments, it would be very useful to have plots against execution 
time, given that the main motivation for this method is scalability. For 
example, the method outperforms random sampling for small datasets, based on 
number of samples, but what happens when you look at execution time? Given that 
random sampling is very cheap, I imagine that it probably does better. Also, as 
mentioned earlier, I would like to see at least one experiment using a big 
dataset, where the method outperforms random sampling, as I am not currently 
convinced of its usefulness.
Also, you present a lot of results and list observations but I felt there was 
not much discussion as to why you observe/obtain some of the results. Given that 
your method is not working very well for CIFAR, I would like to see a more 
thorough investigation as to why that may be the case. This investigation could 
conclude with some "tips" on when it may be a good idea to use your method over 
GAAL, or uncertainty sampling, for example.
Regarding the experimental setup, I find lots of configuration choices very 
arbitrary and have difficulty understanding how they were chosen. For example:
  - For the two-class MNIST you use classes "5" and "7" and for the two-class 
    CIFAR you use classes "automobile" and "horse". Why is that? How did you 
    pick the two classes to use in each case? Do the results match for other 
    class pairs?
  - "learning rate of 0.01 that we decay by a factor of 10 at the 130th and 
    140th epochs" -- end of page 6
  - "In contrast to the previous experiments we use a residual Wasserstein GAN 
    with gradient penalty and soft consistency term" -- page 7 -- why do you 
    make that change?
Questions:
  - Why do you think using Wasserstein GANs perform better than using DCGANs? -- section 5.3.1
  - Why not compare to GAAL in all of figures 3 and 4?
  - How/why were the number of samples you start with and sample in each round, 
    chosen? Do you observe any difference if you increase/decrease the number of 
    samples sampled in each round or if you start with fewer samples?
  - How/why were these model architectures chosen?