Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
Effective convolutional neural networks are trained on large sets of labeled data. However, creating large labeled datasets is a very costly and time-consuming task. Semi-supervised learning uses unlabeled data to train a model with higher accuracy when there is a limited set of labeled data available. In this paper, we consider the problem of semi-supervised learning with convolutional neural networks. Techniques such as randomized data augmentation, dropout and random max-pooling provide better generalization and stability for classifiers that are trained using gradient descent. Multiple passes of an individual sample through the network might lead to different predictions due to the non-deterministic behavior of these techniques. We propose an unsupervised loss function that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. We evaluate the proposed method on several benchmark datasets.
The paper explores the potential of using unlabeled data (transformed versions of the training set) for increasing the generalization properties of convolutional neural network. It formulates an unsupervised loss function, which can be used in conjunction with standard supervised training, for minimizing the variability of multiple passes of the same point through the network due to various stochastic regularization aspects of training (dropout, stochastic pooling etc.) or various augmentation transformations. The loss depends on minimizing the prediction difference among all combinations of transformed or perturbed points (without using label information), along with enforcing mutual exclusivity in the values of the prediction output vector. The large body of this work is of experimental nature, with multiple benchmarks using 2 different network architectures. The unsupervised loss improves performance consistently, especially in the low sample regime, and achieves sate-of-the-art performance in CIFAR10 and CIFAR100.  
This work proposes to use semi-supervised learning, in the form of an unsupervised loss term, for improving the regularization capacity of CNNs. The idea (and the proposed loss) is conceptually simple and enforces stability explicitly by minimizing the difference between predictions corresponding to the same input data point. The paper focuses mainly on the experimental side, devoting the largest part in presenting results when adding the new loss on standard supervised CNNs. This is the stronger aspect of this work, with the weaker being the lack (or the definition) of baselines and the lack of some form of theoretical justification, derivation or discussion.    Novelty/originality: The main contribution is the application of the unsupervised loss term for controlling the stability of the predictions under transformations or stochastic variability. The mutual exclusivity term is from previous work. There is merit in the experimental results promoting the case of semi-supervised learning in training of large neural networks.     Technical quality: Extensive experimental evaluations are given on well known vision datasets with two different models and two different perturbation types (transforming the input and randomization during training), focusing on comparing performance with and without the new loss. Comparisons with one other relevant method is included (but not reproduced or controlled for). Baselines could be further improved (e.g. by including supervised training on augmented labeled data, using the same transformations like in the unsupervised case).  Potential impact: The experimental study makes the case for using unlabeled data to complement supervised training (like in [30]) and enforcing stability, through a simple loss, on the predictions. In addition, results that achieve state-of-the-art are reported on CIFAR10 and CIFAR100.   Q (analysis): The proposed loss is not studied theoretically or discussed in the context of the underlying perturbations. As a matter of fact many of these techniques play the role of additional regularization, while data augmentation acts as an expansion of the training set. For the latter, there is obvious purpose in enforcing the mapping to be the same (same model, perturbed input). The former, however, seems like enforcing different models to give the same prediction, thus canceling out the effects of randomization in the process). Some discussion would be useful here (e.g.see the equivalence of dropout to model averaging).    Q (efficiency): How does the proposed loss scale up with the number of transformations in the augmented set. Since (from (1)), there is a quadratic dependency on the number of augmented versions of the data n, which values of n are practical for computing the loss and being useful for the type of transformations used? The authors use n = 4, 5 here. These values may be small for the case augmentation via geometric transformations (e.g. affine). An empirical study on the role of n (and the trade-off between performance and efficiency) might be insightful.   Q (implementation): How is the loss computed during training? The paper mentions creating min-batches with replicates (or transformations) of the same data points. Will this create gradient bias or affect convergence?     Q (baselines): See below on experiments.   Q: The authors make a comment (Sec. 3, line 102-103) that the loss could be potentially used on labeled samples also to enforce stability. This could potentially be another interesting baseline to explore (in connection to unsupervised vs. supervised training).  Clarity and presentation:  - Paragraph 1 in Sec. 2 (Related work) is very generic and could be revised (or omitted all-together) to focus more on SSL method sin CNNs (like par. 2). Similar, a part of the related work should be devoted to stochastic regularization techniques (pooling, dropout etc.) and data augmentation.   - The network parameter description (line 155) can be confusing (especially if this is typeset in math mode). Is there a way to make the description more compact/descriptive? - Sec 4.2 and 4.3 could be merged (or shortened) as they describe the exact same experimental settings on two different datasets (SVHN and NORB).   Comments on experiments: - Baselines (e.g. line 151/152): The comparisons involve training without augmentation (supervised) and with augmentation for the unsupervised loss (semi-supervised). There should be anther baseline in my opinion -- training on the entire augmented labeled set (supervised with data augmentation). This would be equivalent to adding a loss like \sum_j |y -f(T^j x)| for each x in the training set, thus letting the CNN encode the transformations in the weights.   - Comparisons with Ladder networks [30] (Table 1): The comparison might not be exactly fair since the results are directly taken from Table 2 in [30] (i.e. not reproduced, for example, for the 100 sample training sets or with dropout/fractional max pooling etc.). In addition, the baseline CNN (supervised) for ladder net is different (supervised error is larger). Thus the absolute error comparison here might be misleading --a relative error decrease (with respect to the supervised network used as baseline for each model) might be better.  Similar, the comparison in Table 4 is without data augmentation for the ladder networks.   Secondary/minor comments and typos: - Many references have missing entries or only dates. Please revise. Update ImageNet ref [5].   - line 69: "Xavier's method"?  - line 35: "more stable generalization"?: stability is generalization