One-Shot Unsupervised Cross Domain Translation
Given a single image x from domain A and a set of images from domain B, our task is to generate the analogous of x in B. We argue that this task could be a key AI capability that underlines the ability of cognitive agents to act in the world and present empirical evidence that the existing unsupervised domain translation methods fail on this task. Our method follows a two step process. First, a variational autoencoder for domain B is trained. Then, given the new sample x, we create a variational autoencoder for domain A by adapting the layers that are close to the image in order to directly fit x, and only indirectly adapt the other layers. Our experiments indicate that the new method does as well, when trained on one sample x, as the existing domain transfer methods, when these enjoy a multitude of training samples from domain A. Our code is made publicly available at https://github.com/sagiebenaim/OneShotTranslation.
latent space vectors. In order to encourage generality, we further augment B with samples produced by a slight rotation and with a random horizontal translation.
In the second phase, the variational autoencoder is cloned to create two copies that share the top layers of the encoders and the bottom layers of the decoders, one for the samples in B and one for the sample x in A. The autoencoders are trained with reconstruction losses as well as with a single-sample one-way circularity loss. The samples from domain B continue to train its own copy as in the first step, updating both the shared and the unshared layers. The gradient from sample x updates only the unshared layers and not the shared layers. This way, the autoencoder of B is adjusted by x through the loss incurred on unshared layers for domain B by the circularity loss, and through subsequent adaptation of the shared layers to fit the samples of B. This allows the shared layers to gradually adapt to the new sample x, but prevents overfitting on this single sample. Augmentation is applied, as before, to B and also to x for added stability.
We perform a wide variety of experiments and demonstrate that OST outperforms the existing algorithms in the low-shot scenario. On most datasets the method also presents a comparable accuracy with a single training example to the accuracy obtained by the other methods for the entire set of domain A images. This success sheds new light on the potential mechanisms that underlie unsupervised domain translation, since in the one-shot case, constraints on the inter-sample correlations in domain A do not apply.
In overall, I think this paper proposes a well-designed two steps learning pipeline for one-shot unsupervised image translation. But the explanations about selective backpropagation in the rebuttal are still not clear to me. According to Eq.8-14, it seems that G^S and E^S are not updated in phase II. But according to the Tab. 1 and the rebuttal, they seem to be selectively updated. I strongly suggest the authors to explain the details and motivation in the method part if this paper is accepted. I keep my initial rating. =========================================== This paper proposes a two steps method for one-shot unsupervised image-to-image translation task. This method can help to enable the one-shot unsupervised image translation and also improve the performance continuously given more training samples.  Pros: * The proposed idea is interesting that designing a two steps learning pipeline based on the shared latent space assumption which is implemented with the simple weight sharing technique. * The writing is easy to follow, and empirical experiments show that it is able to translate the image in one-shot setting.  Cons: * If I understand correctly, the shared layers of G^S and E^S will only be updated with loss L^I in phase two of training. The explanation for such selective backpropagation in line 146-148 is not so convincing to me.  ** I would like the authors to explain the function and motivation for such selective backpropagation in more details, since it is the key for the one-shot unsupervised image-to-image translation. ** Does the training of G^S and E^S aim at keeping the shared latent constraint?  ** Why only use one-way cycle loss? I would like the authors to explain the reason of not using two-way cycle loss and show the comparison.  * In the results of style transfer and drawing tasks, the authors only show the OST 1-shot results. I wonder how the proposed method will perform on these tasks with all-shot setting, since for MNIST to SVHN Translation OST is able to perform better than CycleGAN and UNIT when more samples are presented.