Label Efficient Learning of Transferable Representations across Domains and Tasks
We propose a framework that learns a representation transferable across different domains and tasks in a label efficient manner. Our approach battles domain shift with a domain adversarial loss, and generalizes the embedding to novel task using a metric learning-based approach. Our model is simultaneously optimized on labeled source data and unlabeled or sparsely labeled data in the target domain. Our method shows compelling results on novel classes within a new domain even when only a few labeled examples per class are available, outperforming the prevalent fine-tuning approach. In addition, we demonstrate the effectiveness of our framework on the transfer learning task from image object recognition to video action recognition.
The works proposed a new framework to learn a representation that can transfer across domains as well as different tasks. There are two main novelties of this work: 1) a multi-layer domain adversarial loss vs existing work which learns a layer-by-layer embedding (generator) and discriminator; 2) a semantic transfer loss defined in equation (4) to address the cases when the source and target domain have different tasks (classes). Experimental results demonstrated that the multi-layer domain adversarial loss improves domain alignment as well as stability of adversarial training. Table 1 suggests that semantic transfer offers a significant boost to the performance of the proposed method. 
The paper is mostly well-written and easy to follow. The notations are not very consistent. Section 3.3 could be improved. The experimental results showed significant improvement over existing methods with the two modification introduced. There are three components introduced in the semantic transfer section. It is not clear from the experimental results that which one is contributing to the improvement mostly. It is not clear why the authors refer to the loss in equation (5) as a metric-based loss. 
In section 3.3, it seems like the similarity \phi(,) should be defined over the learned embedding instead of the original raw input? Please clarify the notation in section 3.3 to be consistent with section 3.2 if it is the case. 
In equation (4) and (6), do the authors use all the labeled examples or only the prototype examples to compute the cross-entropy? The former case seems prohibitively expensive, especially when the number of samples in the source domain is big.  If it's the later case, how were the prototype examples chosen? 
Minor question:
1) what is the \gamma parameter shown in table 1 ?