Co-regularized Alignment for Unsupervised Domain Adaptation
Deep neural networks, trained with large amount of labeled data, can fail to generalize well when tested with examples from a target domain whose distribution differs from the training data distribution, referred as the source domain. It can be expensive or even infeasible to obtain required amount of labeled data in all possible domains. Unsupervised domain adaptation sets out to address this problem, aiming to learn a good predictive model for the target domain using labeled examples from the source domain but only unlabeled examples from the target domain. Domain alignment approaches this problem by matching the source and target feature distributions, and has been used as a key component in many state-of-the-art domain adaptation methods. However, matching the marginal feature distributions does not guarantee that the corresponding class conditional distributions will be aligned across the two domains. We propose co-regularized domain alignment for unsupervised domain adaptation, which constructs multiple diverse feature spaces and aligns source and target distributions in each of them individually, while encouraging that alignments agree with each other with regard to the class predictions on the unlabeled target examples. The proposed method is generic and can be used to improve any domain adaptation method which uses domain alignment. We instantiate it in the context of a recent state-of-the-art method and observe that it provides significant performance improvements on several domain adaptation benchmarks.
This paper presents a co regularization approach to domain adaptation. They build upon the "VADA" framework by A dirt-t approach to unsupervised domain adaptation. In particular, the VADA framework trains feature extractor 'g' and classifier 'h' by minimizing: distributional distance between features extracted from the source and target domains, label loss on source domain, and entropy + a smoothness operator over the output function (g*h) on the source and target domains.   The contribution of this paper is to first point out that alignment between source and target distributions is possible even with a 'g' that does not align the class distributions across source and target, and vice versa. To this end, the paper proposes to use co-regularization: learn two feature extractors g1 and g2 and two classifiers h1 and h2, via the VADA objective function, while also minimizing the distance between g1*h1 and g2*h2 and maximizing the distance between g1 and g2 for diversity. The authors claim that this can help rule out incorrect alignments.   Cons: 1. This paper argues in section 1 and figure 1 how alignment is not guaranteed to reduce target error and vice versa. However, with their co-regularization framework, they do not rigorously show (either theoretically or with controlled empirical results) that their approach indeed results in better alignment. The arguments on lines 112-116 and 123-128 are quite hand-wavy. This is indeed my biggest gripe with the paper because it is not quite clear how is the co-regularization actually helping in this case. In the co-regularization paper by Rosenberg and Bartlett, the use of two separate RKHS was shown to reduce Rademacher complexity, but what exactly is going on in this case?  2. More of a comment on above than a con: The experimental results show that even with \lambda_div=0, their framework provides improvements. The diversity in g1 and g2, in this case, is only coming from randomness (as the authors themselves point out on lines 242-245). In effect, this relies on the randomness of initialization and training along with non-convexity of the objective function to generate diversity. This needs to be explained and investigated further -- can we just rule out more and more "incorrect alignments" by randomly generating classifiers. If so, can we just make the feature generator better by learning 3 instead of 2 classifiers?  Question; 1. What set of parameters are finally used for testing on the target layer?   After author feedback: Upgraded my score to 6 provided the authors add their kNN explanation to the paper.