We study the joint distribution matching problem which aims at learning bidirectional mappings to match the joint distribution of two domains. This problem occurs in unsupervised image-to-image translation and video-to-video synthesis tasks, which, however, has two critical challenges: (i) it is difficult to exploit sufficient information from the joint distribution; (ii) how to theoretically and experimentally evaluate the generalization performance remains an open question. To address the above challenges, we propose a new optimization problem and design a novel Joint Wasserstein Auto-Encoders (JWAE) to minimize the Wasserstein distance of the joint distributions in two domains. We theoretically prove that the generalization ability of the proposed method can be guaranteed by minimizing the Wasserstein distance of joint distributions. To verify the generalization ability, we apply our method to unsupervised video-to-video synthesis by performing video frame interpolation and producing visually smooth videos in two domains, simultaneously. Both qualitative and quantitative comparisons demonstrate the superiority of our method over several state-of-the-arts.
This paper studies the joint distribution matching problem where given data samples in two different domains, one is interested in learning a bi-directional mapping between unpaired data elements in these domains. The paper proposes a joint Wasserstein auto-encoder (JWAE) to solve this problem. The paper shows that under the decomposable cost metric and deterministic decoding maps, the optimization problem associated with the JWAE formulation can be reduced to a tractable optimization problem. The paper also establishes a generalization bound for the JWAE formulation. Finally, the paper conducts an experimental evaluation of the proposed solution with the help of a video-to-video synthesis problem and show improved performance as compared to the existing results in the literature.
Overall, the reviewer finds that the paper considers an important problem and proposes some interesting ideas to tackle the problem. However, in its current form, there is a large scope for improvement in the presentation of the paper. The paper is full of errors/typos which make it an extremely difficult read (see my comments below). That said the paper fairs quite well as compared to other existing methods. Since the reviewer is not very much familiar with this field, the reviewer leaves it to the other reviewers to decide the significance of these results.
Pros: 
1) The paper aims to provide a theoretical treatment of the joint distribution matching problem which has many interesting applications, including image-to-image translation and video-to-video synthesis. 
2) The proposed method in the paper had good empirical performance on the real world datasets.
Cons:
The paper is very poorly written with many typos and (possibly) mistakes. Some of the comments in this direction are as follows.
1) The paper does not formally define the underlying problem before diving into the details of the proposed solution. The authors only informally talk about the problem in the introduction. Given that the ICLR has a wide audience, it would have been nice if the authors have made the presentation of the paper self-contained.
2) In the same vein, the paper talks about many important quantities without introducing them first. E.g., what are $E_{A}(f^*)$, $E_B^g(f^*)$ etc. in the statement of Theorem 2? These quantities are first defined inside the proofs in the supplementary material!
3) Some of the notation in the paper is also very confusing. For example, cross-domain mapping have two different sets of notations. $(E1oG2, E2oG)$ in Sec. 4.2 and $(G2oE1, G1oE2)$ in Section 5. It should be latter. Similarly, Sec. 4.2 refers to $E1oG1$ and $E2oG2$ as auto-encoders, which should be $G1oE1$ and $G2oE2$, respectively. In Sec. 3, the authors refer to $N$ and $M$ as the number of samples in the $X$ and $Y$ domain, respectively. This is then reversed in Theorem 2 and 4. These are only a small list of large number of such typos. Also, what is the notion defined in the last line of Sec. 3?
4) It is not clear to me why the sets $Q_1$ and $Q_2$ in Theorem 1 are define in their current forms. In particular, it is not clear why the equality hold in Eq. (17) in the proof of Theorem 1. 
5) One line in the proof of Lemma 1 says, "Specifically, we choose its equality, then we have". Could the authors elaborate on this?
6) Eq. (24) should be inequality?
7) Given that the authors write a regularized problem in (4). Does that mean now sets $Q_1$ and $Q_2$ are different from how they are defined in the statement of Theorem 1?
#########################
Post rebuttal: The authors have addressed most of my concerns regarding the poor presentation of the earlier version. I have updated my score.