UNSUPERVISED DOMAIN ADAPTATION FOR DISTANCE METRIC LEARNING
Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.
The authors studied an interesting problem of unsupervised domain adaptation when the source and the target domains have disjoin labels spaces. The paper proposed a novel feature transfer network, that optimizes domain adversarial loss and domain separation loss.
Strengths:
1) The proposed approach on Feature Transfer Network was novel and interesting.
2) The paper was very well written with a good analysis of various choices.
3) Extensive empirical analysis on multi-class settings with a traditional MNIST dataset and a real-world face recognition dataset. 
Weakness:
1) Practical considerations addressing feature reconstruction loss needs more explanation.
Comments:
The technical contribution of the paper was sound and novel. The paper considered existing work and in a good way generalizes and extends into disjoint label spaces. It was easy to read and follow, most parts of the paper including the Appendix make it a good contribution. However, the reviewer has the following suggestions" 
1. Under the practical considerations for preventing the mode collapse via feature reconstruction, how is the reference network trained? In the Equation(6) for feature reconstruction, the f_ref term maps the source and target domain examples to new feature space. What do you mean by references network trained on the label data? Please clarify.
2. Under the practical considerations for replacing the verification loss, it is said that "Our theoretical analysis suggests to use a verification
the loss that compares the similarity between a pair of images" - Can you please cite the references to make it easier for the reader to follow.