A DIRT-T APPROACH TO UNSUPERVISED DOMAIN ADAPTATION
Domain adaptation refers to the problem of leveraging labeled data in a source domain to learn an accurate model in a target domain where labels are scarce or unavailable. A recent approach for finding a common representation of the two domains is via domain adversarial training (Ganin & Lempitsky, 2015), which attempts to induce a feature extractor that matches the source and target feature distributions in some feature space. However, domain adversarial training faces two critical limitations: 1) if the feature extraction function has high-capacity, then feature distribution matching is a weak constraint, 2) in non-conservative domain adaptation (where no single classifier can perform well in both the source and target domains), training the model to do well on the source domain hurts performance on the target domain. In this paper, we address these issues through the lens of the cluster assumption, i.e., decision boundaries should not cross high-density data regions. We propose two novel and related models: 1) the Virtual Adversarial Domain Adaptation (VADA) model, which combines domain adversarial training with a penalty term that punishes violation of the cluster assumption; 2) the Decision-boundary Iterative Refinement Training with a Teacher (DIRT-T) 1 model, which takes the VADA model as initialization and employs natural gradient steps to further minimize the cluster assumption violation. Extensive empirical results demonstrate that the combination of these two models significantly improve the state-of-the-art performance on the digit, traffic sign, and Wi-Fi recognition domain adaptation benchmarks.
As there are many kinds of domain adaptation problems, the need to mix several learning strategies to improve the existing approaches is obvious. However, this task is not necessarily easy to succeed. The authors proposed a sound approach to learn a proper representation (in an adversarial way) and comply the cluster assumption.
The experiments show that this Virtual Adversarial Domain Adaptation network (VADA) achieves great results when compared to existing learning algorithms. Moreover, we also see the learned model is consistently improved using the proposed "Decision-boundary Iterative Refinement Training with a Teacher" (DIRT-T) approach.
The proposed methodology relies on multiple choices that could sometimes be better studied and/or explained. Namely, I would like to empirically see which role of the locally-Lipschitz regularization term (Equation 7). Also, I wonder why this term is tuned by an hyperparameter (lamda_s) for the source, while a single hyperparamer (lambda_t) is used for the sum of the two target quantity.
 
On the theoretical side, the discussion could be improved. Namely, Section 3 about "limitation of domain adversarial training" correctly explained that "domain adversarial training may not be sufficient for domain adaptation if the feature extraction function has high-capacity". It would be interesting to explain whether this observation is consistent with Theorem 1 of the paper (due to Ben-David et al., 2010), on which several domain adversarial approaches are based. The need to consider supplementary assumptions (such as ) to achieve good adaptation can also be studied through the lens of more recent Ben-David's work, e.g. Ben-David and Urner (2014). In the latter, the notion of "Probabilistic Lipschitzness", which is a relaxation of the "cluster assumption" seems very related to the actual work.
Reference:
Ben-David and Urner. Domain adaptation-can quantity compensate for quality?, Ann. Math. Artif. Intell., 2014
Pros:
- Propose a sound approach to mix two complementary strategies for domain adaptation.
- Great empirical results.
Cons:
- Some choices leading to the optimization problem are not sufficiently explained.
- The theoretical discussion could be improved.
Typos:
- Equation 14: In the first term (target loss), theta should have an index t (I think).
- Bottom of page 6: "... and that as our validation set" (missing word).