AUGMENTED CYCLIC ADVERSARIAL LEARNING FOR LOW RESOURCE DOMAIN ADAPTATION
Training a model to perform a task typically requires a large amount of data from the domains in which the task will be applied. However, it is often the case that data are abundant in some domains but scarce in others. Domain adaptation deals with the challenge of adapting a model trained from a data-rich source domain to perform well in a data-poor target domain. In general, this requires learning plausible mappings between domains. CycleGAN is a powerful framework that efficiently learns to map inputs from one domain to another using adversarial training and a cycle-consistency constraint. However, the conventional approach of enforcing cycle-consistency via reconstruction may be overly restrictive in cases where one or more domains have limited training data. In this paper, we propose an augmented cyclic adversarial learning model that enforces the cycle-consistency constraint via an external task specific model, which encourages the preservation of task-relevant content as opposed to exact reconstruction. We explore digit classification in a low-resource setting in supervised, semi and unsupervised situation, as well as high resource unsupervised. In low-resource supervised setting, the results show that our approach improves absolute performance by 14% and 4% when adapting SVHN to MNIST and vice versa, respectively, which outperforms unsupervised domain adaptation methods that require high-resource unlabeled target domain. Moreover, using only few unsupervised target data, our approach can still outperforms many high-resource unsupervised models. Our model also outperforms on USPS to MNIST and synthetic digit to SVHN for high resource unsupervised adaptation. In speech domains, we similarly adopt a speech recognition model from each domain as the task specific model. Our approach improves absolute performance of speech recognition by 2% for female speakers in the TIMIT dataset, where the majority of training samples are from male voices.
The authors propose an extension of cycle-consistent adversarial adaptation methods in order to tackle domain adaptation in settings where a limited amount of supervised target data is available (though they also validate their model in the standard unsupervised setting as well). The method appears to be a natural generalization/extension of CycleGAN/CyCADA. It uses the ideas of the semantic consistency loss and training on adapted data from CyCADA, but "fills out" the model by applying these techniques in both directions (whereas CyCADA only applied them in the source-to-target direction).
The writing in this paper is a little awkward at times (many omitted articles such as "the" or "a'), but, with a few exceptions, it is generally easy to understand what the authors are saying. They provide experiments in a variety of settings in order to validate their model, including both visual domain adaptation and speech domain adaptation. The experiments show that their model is effective both in low-resource supervised adaptation settings as well as high-resource unsupervised adaptation settings. An ablation study, provided in Section 4.1, helps to understand how well the various instantiations of the authors' model perform, indicating that enforcing consistency in both methods is crucial to achieving performance beyond the simple baselines.
It's a little hard to understand how this method stands in comparison to existing work. Table 3 helps to show that the model can scale up to the high-resource setting, but it would also be nice to see the reverse: comparisons against existing work run in the limited data setting, to better understand how much limited data negatively impacts the performance of models that weren't designed with this setting in mind.
I would've also liked to see more comparisons against the simple baseline of a classifier trained exclusively on the available supervised target data, or with the source and target data together—in my experience, these baselines can prove to be surprisingly strong, and would give a better sense of how effective this paper's contributions are. This corresponds to rows 2 and 3 of Table 1, and inspection of the numbers in that table shows that the baseline performance is quite strong even relative to the proposed method, so it would be nice to see these numbers in Table 2 as well, since that table is intended to demonstrate the model's effectiveness across a variety of different domain shifts.
While it's nice that the model is experimentally validated on the speech domain, the experiment itself is not explained well. The speech experiments are hard to understand—it's unclear what the various training sets are, such as "Adapted Male" or "All Data," making it hard to understand exactly what numbers should be compared. Why is there no CycleGAN result for "Female + Adapted Male," or "All Data + Adapted Male," for example? The paper would greatly benefit from a more careful explanation and analysis of this experimental setting.
Ultimately, I think the idea is a nice generalization of previous work, and the experiments seem to indicate that the model is effective, but the limited scope of the experiments prevent me from being entirely convinced. The inclusion of additional baselines and a great deal of clarification on the speech experiments would improve the quality of this paper enormously.
---
Update: After looking over the additional revisions and experiments, I'm bumping this to a weak accept. I agree with reviewer 3 that novelty is not the greatest, but there is a useful contribution here, and the demonstration of its effectiveness on low resource settings is valuable, since in a practical setting it is usually feasible to manually label a few examples.
I'm still not convinced by the TIMIT experiments, now that I better understand them, since the F+M baseline is quite strong and very simple to run. It simply doesn't seem worthwhile to introduce all of this extra machinery for such a marginal improvement, but the experiment does serve the job of at least demonstrating an improvement over existing methods.