Published as a conference paper at ICLR 2018 IDENTIFYING ANALOGIES ACROSS DOMAINS
Identifying analogies across domains without supervision is an important task for artificial intelligence. Recent advances in cross domain image mapping have concentrated on translating images across domains. Although the progress made is impressive, the visual fidelity many times does not suffice for identifying the matching sample from the other domain. In this paper, we tackle this very task of finding exact analogies between datasets i.e. for every image from domain A find an analogous image in domain B. We present a matching-by-synthesis approach: AN-GAN, and show that it outperforms current techniques. We further show that the cross-domain mapping task can be broken into two parts: domain alignment and learning the mapping function. The tasks can be iteratively solved, and as the alignment is improved, the unsupervised translation function reaches quality comparable to full supervision.
This paper adds an interesting twist on top of recent unpaired image translation work. A domain-level translation function is jointly optimized with an instance-level matching objective. This yields the ability to extract corresponding image pairs out of two unpaired datasets, and also to potentially refine unpaired translation by subsequently training a paired translation function on the discovered matches. I think this is a promising direction, but the current paper has unconvincing results, and it’s not clear if the method is really solving an important problem yet.
My main criticism is with the experiments and results. The experiments focus almost entirely on the setting where there actually exist exact matches between the two image sets. Even the partial matching experiments in Section 4.1.2 only quantify performance on the images that have exact matches. This is a major limitation since the compelling use cases of the method are in scenarios where we do not have exact matches. It feels rather contrived to focus so much on the datasets with exact matches since, 1) these datasets actually come as paired data and, in actual practice, supervised translation can be run directly, 2) it’s hard to imagine datasets that have exact but unknown matches (I welcome the authors to put forward some such scenarios), 3) when exact matches exist, simpler methods may be sufficient, such as matching edges. There is no comparison to any such simple baselines.
I think finding analogies that are not exact matches is much more compelling. Quantifying performance in this case may be hard, and the current paper only offers a few qualitative results. I’d like to see far more results, and some attempt at a metric. One option would be to run user studies where humans judge the quality of the matches. The results shown in Figure 2 don’t convince me, not just because they are qualitative and few, but also because I’m not sure I even agree that the proposed method is producing better results: for example, the DiscoGAN results have some artifacts but capture the texture better in row 3.
I was also not convinced by the supervised second step in Section 4.3. Given that the first step achieves 97% alignment accuracy, it’s no surprised that running an off-the-shelf supervised method on top of this will match the performance of running on 100% correct data. In other words, this section does not really add much new information beyond what we could already infer given that the first stage alignment was so successful.
What I think would be really interesting is if the method can improve performance on datasets that actually do not have ground truth exact matches. For example, the shoes and handbags dataset or even better, domain adaptation datasets like sim to real.
I’d like to see more discussion of why the second stage supervised problem is beneficial. Would it not be sufficient to iterate alpha and T iterations enough times until alpha is one-hot and T is simply training against a supervised objective (Equation 7)?
Minor comments:
1. In the intro, it would be useful to have a clear definition of “analogy” for the present context.
2. Page 2: a link should be provided for the Putin example, as it is not actually in Zhu et al. 2017.
3. Page 3: “Weakly Supervised Mapping” — I wouldn’t call this weakly supervised. Rather, I’d say it’s just another constraint / prior, similar to cycle-consistency, which was referred to under the “Unsupervised” section.
4. Page 4 and throughout: It’s hard to follow which variables are being optimized over when. For example, in Eqn. 7, it would be clearer to write out the min over optimization variables.
5. Page 6: The Maps dataset was introduced in Isola et al. 2017, not Zhu et al. 2017.
6. Page 7: The following sentence is confusing and should be clarified: “This shows that the distribution matching is able to map source images that are semantically similar in the target domain.”
7. Page 7: “This shows that a good initialization is important for this task.” — Isn’t this more than initialization? Rather, removing the distributional and cycle constraints changes the overall objective being optimized.
8. In Figure 2, are the outputs the matched training images, or are they outputs of the translation function?
9. Throughout the paper, some citations are missing enclosing parentheses.