Joint distribution optimal transportation for domain adaptation
This paper deals with the unsupervised domain adaptation problem, where one wants to estimate a prediction function f in a given target domain without any labeled sample by exploiting the knowledge available from a source domain where labels are known. Our work makes the following assumption: there exists a nonlinear transformation between the joint feature/label space distributions of the two domain P s and P t that can be estimated with optimal transport. We propose a solution of this problem that allows to recover an estimated target P f t = (X, f (X)) by optimizing simultaneously the optimal coupling and f . We show that our method corresponds to the minimization of a bound on the target error, and provide an efficient algorithmic solution, for which convergence is proved. The versatility of our approach, both in terms of class of hypothesis or loss functions is demonstrated with real world classification and regression problems, for which we reach or surpass state-of-the-art results.
The manuscript introduces a new technique for unsupervised domain adaptation (DA) based on optimal transport (OT). Previous related techniques (that are state-of-the-art) use optimal transport to find a mapping between the marginal distributions of source and target.  
The manuscript proposes to transports the joint distributions instead of the marginals. As the target does not have labels, the learned prediction function on the target is used as proxy. The consequence is an iterative algorithm that alternates between 1) estimating the transport for fixed data points and labeling, and 2) estimating the labeling function for fixed transport. As such, the algorithm can be seen as a self-learning inspired extension of previous work, which executed 1) and 2) only once. 
In addition to the algorithm, the manuscript contains a high probability generalization bound, similar to the discrepancy-based bounds of Ben-David et al [24]. It is used as justification that minimizing the joint OT distance should result in better predictions. Formally, this is not valid, however, because the bound is not uniform in the labeling function, so it might not hold for the result of the minimization. 
There are also experiments on three standard datasets that show that the proposed method achieve state-of-the-art performance (though the differences to the previous OT based methods are small). 
strength:
- the paper is well written, the idea makes sense
- the method seems practical, as both repeated steps resemble previous OT-based DA work
- the theory seems correct (but is not enough to truly justify the method, see below)
- the experiments are done well, including proper model selection (which is rare in DA)
- the experimental results are state-of-the-art
weaknesses:
- the manuscript is mainly a continuation of previous work on OT-based DA
- while the derivations are different, the conceptual difference is previous work is limited 
- theoretical results and derivations are w.r.t. the loss function used for learning (e.g. 
  hinge loss), which is typically just a surrogate, while the real performance measure would 
  be 0/1 loss. This also makes it hard to compare the bounds to previous work that used 0-1 loss 
- the theorem assumes a form of probabilistic Lipschitzness, which is not explored well. 
  Previous discrepancy-based DA theory does not need Prob.Lipschitzness and is more flexible 
  in this respect.
- the proved bound (Theorem 3.1) is not uniform w.r.t. the labeling function $f$. Therefore, 
  it does not suffice as a justification for the proposed minimization procedure. 
- the experimental results do not show much better results than previous OT-based DA methods
- as the proposed method is essentially a repeated application of the previous work, I would have
  hoped to see real-data experiments exploring this. Currently, performance after different number
  of alternating steps is reported only in the supplemental material on synthetic data.
- the supplemental material feels rushed in some places. E.g. in the proof of Theorem 3.1, the 
  first inequality on page 4 seems incorrect (as the integral is w.r.t. a signed measure, not a 
  prob.distr.). I believe the proof can be fixed, though, because the relation holds without
  absolute values, and it's not necessary to introduce these in (3) anyway.  
- In the same proof, Equations (7)/(8) seem identical to (9)/(10)
questions to the authors:
- please comment if the effect of multiple BCD on real data is similar to the synthetic case 
***************************
I read the author response and I am still in favor of accepting the work.