Introspective Classification with Convolutional Nets
We propose introspective convolutional networks (ICN) that emphasize the importance of having convolutional neural networks empowered with generative capabilities. We employ a reclassification-by-synthesis algorithm to perform training using a formulation stemmed from the Bayes theory. Our ICN tries to iteratively: (1) synthesize pseudo-negative samples; and (2) enhance itself by improving the classification. The single CNN classifier learned is at the same time generative -being able to directly synthesize new samples within its own discriminative model. We conduct experiments on benchmark datasets including MNIST, CIFAR-10, and SVHN using state-of-the-art CNN architectures, and observe improved classification results.
The paper proposes a technique to improve the test accuracy of a discriminative model, by synthesizing additional negative input examples during the training process of the model. The negative example generation process has a Bayesian motivation, and is realized by "optimizing" for images (starting from random Gaussian noise) to maximize the probability of a given class label, a la DeepDream or Neural Artistic Style. These generated examples are added to the training set, and training is halted based on performance on a validation set. Experiments demonstrate that this procedure yields (very modest) improvements in test accuracy, and additionally provides some robustness against adversarial examples.
The core idea is quite elegant, with an intuitive picture of using the "hard" negatives generated by the network to tighten the decision boundaries around the positive examples. The authors provide some theoretical justification showing that the distribution of these negatives examples converges to the distribution of positive examples.
While I find the approach intriguing, there are a few shortcomings. The improvement in classification accuracy demonstrated in the experiments is relatively small for the datasets shown, although the MNIST demonstration with varying sample sizes is very encouraging. A more complex dataset (e.g. ImageNet) would be very interesting to see, but potentially quite expensive to train... I'm unable to determine how many pseudo-negative samples are generated in each round (it says "e.g. 200" at one point; is it 200?), which affects the practicality of the approach (how much slower is training?).
While the Bayesian motivation is nice, it's not clear to me exactly what it means to sample from the distribution of negative examples using SGD. I'm not actively familiar with the references listed tying SGD to MCMC; if there is in fact a meaningful interpretation of the proposed sampling procedure that justifies the vague (but repeated) use of the term "fair sample", it would be nice to include a more explicit summary in the text.
Finally, it would have been nice to see comparisons against DCGAN for CIFAR10 and SVHN as well, as well as a simple baseline of label smoothing (instead of using one-hot target labels, use a mixture of e.g. .9 * one-hot + .1 * uniform). Label smoothing has been reported (e.g., in https://arxiv.org/pdf/1512.00567.pdf) to yield comparably sized improvements, essentially by encouraging the model to be less confident. One explanation for the success of the ICN approach could simply be that by training on negative samples that are close to the positive samples, the model is forced to be less confident overall, reducing overfitting. Comparing against label smoothing could help tease out whether that explanation is sufficient (and if so, label smoothing is substantially simpler and cheaper, if a bit of a hack).
Despite the above criticisms, I think the paper is well written and could be a valuable contribution; I recommend acceptance.