Algorithms and Theory for Multiple-Source Adaptation
We present a number of novel contributions to the multiple-source adaptation problem. We derive new normalized solutions with strong theoretical guarantees for the cross-entropy loss and other similar losses. We also provide new guarantees that hold in the case where the conditional probabilities for the source domains are distinct. Moreover, we give new algorithms for determining the distributionweighted combination solution for the cross-entropy loss and other losses. We report the results of a series of experiments with real-world datasets. We find that our algorithm outperforms competing approaches by producing a single robust model that performs well on any target mixture distribution. Altogether, our theory, algorithms, and empirical results provide a full solution for the multiple-source adaptation problem with very practical benefits.
Summary of the paper:  This paper is interested in the problem of multiple-source adaptation. More precisely it is interested in the setting where a good model and an estimation of the distribution is available for several source domains. The goal is then to learn a good combination of the models to accurately classify target examples drawn from a mixture of the source domains. On the one hand the paper extends the theoretical results of [Mansour et al., 2008] to the stochastic setting where there exists a distribution over the joint input-output space. On the other hand it proposes an algorithm to correctly combine the models available from the different domains. The performance of the approach is empirically demonstrated on several tasks.  Main comments:  Pros:  - The paper is very well written and easy to read  - The setting considered is more general than existing works  - The proposed algorithm to estimate the target mixture is new, theoretically sound (Lemma 4) and works well in practice   Cons:  - The theoretical results are a bit incremental [Mansour et al. 2008]  - The practical usefulness of the method is not so clear as it requires an estimate of the distribution and a model for each domain  Detailed comments:  This paper is well written and, despite being quite technical, is easy to follow.  In the experiment displayed in Figure 1 and introduced Lines 246 to 251, I would have appreciated to see the results of the predictor used in Mansour et al. [2008]. Even it is not a real solution, it remains a good baseline to assess the limit of the method considered, that is it is probably one of the best achievable results using distribution-weighted combination of the source models.  The theoretical results are largely based on the previous work of Mansour et al. [2008] and mainly consist of an extension from the deterministic to the stochastic setting. In particular it considers a more realistic setting where the conditional probabilities of each output given an input might be different for the source and the target domain.  One of the main drawbacks of the proposed approach is that it assumes that the underlying distribution of the inputs is available for each domain which might not be very realistic in practice. This issue can be overcome by using a large number of unlabelled data from each domain to estimate the underlying distribution. In which kind of applications would this kind of setting (a good model and a set of unlabelled examples for each domain) be reasonable?  Some other minor comments:  - In the main paper, Line 111: I would have appreciated to have a bit more intuition about the meaning of \epsilon_T and not a simple definition, that is what does it represent or account for?  - In the supplementary, in the inequality following Line 440: It should be \hat{h} rather than h.  After the rebuttal:  The rebuttal solved my concerns regarding the practical usefulness of the method. However I still think that the contribution is a bit incremental and I agree with Reviewer 1 that a more extensive empirical evaluation would have been beneficial.