Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used.
The paper proposes to substitute simple noising operations with many data augmentation methods in consistency-based semi-supervised learning. The main idea is the same as previous work: constrain the model predictions of unlabeled examples to be invariant to different noise. The proposed UDA is evaluated on a wide range of language and vision tasks.
Overall, the paper is well-written and clear. The most impressive point of this paper is its strong empirical results. However, it looks not surprising to me that more data augmentations found in supervised learning are also effective in semi-supervised learning. The paper fails to provide any theoretical insights but a thorough empirical evaluation.
One of my concerns is that the hyperparameters on vision tasks follow those of AutoAugment, which is carefully tuned on supervised tasks. Apparently, their hyperparameters are based on the whole labeled training dataset. In this case, the adopted hyperparameters include sort of information of the whole labeled dataset. Is it fair?
Another concern is how to control the strength of augmentations. For example, for digit images like SVHN, a "6" rotates by 180 degree is "9", whose prediction should change correspondingly. In this case, the assumption of invariance does not hold when the augmentation is too strong. 
I'm willing to increase my score if the authors address my concerns.