Domain adaptation is critical for success in new, unseen environments. Adversarial adaptation models applied in feature spaces discover domain invariant representations, but are difficult to visualize and sometimes fail to capture pixel-level and low-level domain shifts. Recent work has shown that generative adversarial networks combined with cycle-consistency constraints are surprisingly effective at mapping images between domains, even without the use of aligned image pairs. We propose a novel discriminatively-trained Cycle-Consistent Adversarial Domain Adaptation model. CyCADA adapts representations at both the pixel-level and feature-level, enforces cycle-consistency while leveraging a task loss, and does not require aligned pairs. Our model can be applied in a variety of visual recognition and prediction settings. We show new state-of-the-art results across multiple adaptation tasks, including digit classification and semantic segmentation of road scenes demonstrating transfer from synthetic to real world domains.
This paper essentially uses CycleGANs for Domain Adaptation. My biggest concern is that it doesn't adequately compare to similar papers that perform adaptation at the pixel level (eg. Shrivastava et al-'Learning from Simulated and Unsupervised Images through Adversarial Training' and Bousmalis et al - 'Unsupervised Pixel-level Domain Adaptation with GANs', two similar papers published in CVPR 2017 -the first one was even a best paper- and available on arXiv since December 2016-before CycleGANs). I believe the authors should have at least done an ablation study to see if the cycle-consistency loss truly makes a difference on top of these works-that would be the biggest selling point of this paper. The experimental section had many experiments, which is great. However I think for semantic segmentation it would be very interesting to see whether using the adapted synthetic GTA5 samples would improve the SOTA on Cityscapes. It wouldn't be unsupervised domain adaptation, but it would be very impactful. Finally I'm not sure the oracle (train on target) mIoU on Table 2 is SOTA, and I believe the proposed model's performance is really far from SOTA.
Pros:
* CycleGANs for domain adaptation! Great idea!
* I really like the work on semantic segmentation, I think this is a very important direction
Cons:
* I don't think Domain separation networks is a pixel-level transformation-that's a feature-level transformation, you probably mean to use Bousmalis et al. 2017. Also Shrivastava et al is missing from the image-level papers.
* the authors claim that Bousmalis et al, Liu & Tuzel and Shrivastava et al ahve only been shown to work for small image sizes. There's a recent work by Bousmalis et al. (Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping) that shows these methods working well (w/o cycle-consistency) for settings similar to semantic segmentation at a relatively high resolution. Also it was mentioned that these methods do not necessarily preserve content, when pixel-da explicitly accounts for that with a task loss (identical to the semantic loss used in this submission)
* The authors talk about the content similarity loss on the foreground in Bousmalis et al. 2017, but they could compare to this method w/o using the content similarity or using a different content similarity tailored to the semantic segmentation tasks, which would be trivial.
* Math seems wrong in (4) and (6). (4) should be probably have a minus instead of a plus. (6) has an argmin of a min, not sure what is being optimized here. In fact, I'm not sure if eg you use the gradients of f_T for training the generators?
* The authors mention that the pixel-da approach cross validates with some labeled data. Although I agree that is not an ideal validation, I'm not sure if it's equivalent or not the authors' validation setting, as they don't describe what that is.
* The authors present the semantic loss as novel, however this is the task loss proposed by the pixel-da paper.
* I didn't understand what pixel-only and feat-only meant in tables 2, 3, 4. I couldn't find an explanation in captions or in text
=====
Post rebuttal comments:
Thanks for adding content in response to my comments. The cycle ablation is still a sticky point for me, and I'm still left not sure if cycle-consistency really offers an improvement. Although I applaud your offering examples of failures for when there's no cycle-consistency, these are circumstantial and not quantitative.  The reader is still left wondering why and when is the cycle-consistency loss is appropriate. As this is the main novelty, I believe this should be in the forefront of the experimental evaluation.