Published as a conference paper at ICLR 2018 SELF-ENSEMBLING FOR VISUAL DOMAIN ADAPTATION
This paper explores the use of self-ensembling for visual domain adaptation problems. Our technique is derived from the mean teacher variant (Tarvainen & Valpola (2017)) of temporal ensembling (Laine & Aila (2017)), a technique that achieved state of the art results in the area of semi-supervised learning. We introduce a number of modifications to their approach for challenging domain adaptation scenarios and evaluate its effectiveness. Our approach achieves state of the art results in a variety of benchmarks, including our winning entry in the VISDA-2017 visual domain adaptation challenge. In small image benchmarks, our algorithm not only outperforms prior art, but can also achieve accuracy that is close to that of a classifier trained in a supervised fashion.
This paper presents a domain adaptation algorithm based on the self-ensembling method proposed by [Tarvainen & Valpola, 2017]. The main idea is to enforce the agreement between the predictions of the teacher and the student classifiers on the target domain samples while training the student to perform well on the source domain. The teacher network is simply an exponential moving average of different versions of the student network over time.   
Pros:
+ The paper is well-written and easy to read
+ The proposed method is a natural extension of the mean teacher semi-supervised learning model by [Tarvainen & Valpola, 2017]
+ The model achieves state-of-the-art results on a range of visual domain adaptation benchmarks (including top performance in the VisDA17 challenge)
Cons:
- The model is tailored to the image domain as it makes heavy use of the data augmentation. That restricts its applicability quite significantly. I’m also very interested to know how the proposed method works when no augmentation is employed (for fair comparison with some of the entries in Table 1).
- I’m not particularly fond of the engineering tricks like confidence thresholding and the class balance loss. They seem to be essential for good performance and thus, in my opinion, reduce the value of the main idea.
- Related to the previous point, the final VisDA17 model seems to be engineered too heavily to work well on a particular dataset. I’m not sure if it provides many interesting insights for the scientific community at large.
In my opinion, it’s a borderline paper. While the best reported quantitative results are quite good, it seems that achieving those requires a significant engineering effort beyond just applying the self-ensembling idea. 
Notes:
* The paper somewhat breaks the anonymity of the authors by mentioning the “winning entry in the VISDA-2017”. Maybe it’s not a big issue but in my opinion it’s better to remove references to the competition entry.
* Page 2, 2.1, line 2, typo: “stanrdard” -> “standard”
Post-rebuttal revision:
After reading the authors' response to my review, I decided to increase the score by 2 points. I appreciate the improvements that were made to the paper but still feel that this work a bit too engineering-heavy, and the title does not fully reflect what's going on in the full pipeline.