Disentangling underlying generative factors of a data distribution is important for interpretability and generalizable representations. In this paper, we introduce two novel disentangling methods. Our first method, Unlabeled Disentangling GAN (UD-GAN, unsupervised), decomposes the latent noise by generating similar/dissimilar image pairs and it learns a distance metric on these pairs with siamese networks and a contrastive loss. This pairwise approach provides consistent representations for similar data points. Our second method (UD-GAN-G, weakly supervised) modifies the UD-GAN with user-defined guidance functions, which restrict the information that goes into the siamese networks. This constraint helps UD-GAN-G to focus on the desired semantic variations in the data. We show that both our methods outperform existing unsupervised approaches in quantitative metrics that measure semantic accuracy of the learned representations. In addition, we illustrate that simple guidance functions we use in UD-GAN-G allow us to directly capture the desired variations in the data.
Summary
The paper presents a novel approach for learning a generative model where different factors of variations can be independently manipulated. The method is build upon  the GAN framework where the latent variables are divided into different subsets (chunks) which are expected to encode information about high-level factors of variation. To this end, a Siamese Network for each chunk is trained with a contrastive loss minimizing the distance between generated images sharing the same factor (the latent variables in the chunk are equal), and maximizing the distance between pairs where the latent variables differ. Given that the proposed model fails in this fully-unsupervised setting, the authors propose to add weak-supervision into the model by forcing the Siamese networks to  focus only on particular aspects of generated images (e.g, color, edges, etc..). This is achieved by applying  a basic transformation  over the input images in order to remove specific information. The evaluation of the  proposed model is carried out using the MS-Celeb dataset where the authors provide qualitative results.
Methodology
*Disentangling generative factors without explicit labels is a challenging and interesting problem. The idea of dividing the latent representation in different subsets and using a proxy task involving triplets of images has been already explored in [3]. However, the use of Siamese networks in this context is novel and sound.
*As shown in the reported results, the proposed method fails to learn meaningful factors in the unsupervised setting. However, the authors do not provide an in-depth discussion of this phenomena. Given that previous works [1,2,3] have successfully addressed this problem using a completely unsupervised approach, it would be necessary to give more insights about: (i) why the proposed method is failing (ii) why this negative result is interesting and (iii) if the method could be useful in other potential scenarios. 
*The strategy proposed to introduce weak-supervision is too ad-hoc. I agree that using cues such as the average color of an image can be useful if we want to model basic factors of variation. However, it is unclear how a similar strategy could be applied if we are interested in learning variables with higher-level semantics such as the expression of a face or its pose.
*As far as I understand, the transformations applied to the input images (e.g, edge detection) must be differentiable (given that it is necessary to backpropagate the gradient of the contrastive loss through the generator network). If this is the case, this should be properly discussed in the paper. Moreover, given that the amount of differentiable transformations is reduced, this also limits the application of the proposed method for more interesting scenarios. 
*It is not clear why the latent variables modelling the generative factors are defined using a Gaussian prior. How the case where two images have a very similar latent factor is avoided while generating pairs of images for the Siamese network?  Have the authors considered to use categorical or binary variables? The use of the contrastive loss sounds more appropriate in this case.
Experimental results
*The experimental section is too limited. First of all, only a small number of qualitative results are reported and, therefore, it is very difficult to assess the proposed method and draw any conclusion. For example, when the edge extractor is used, what kind of information is modeled by the latent variables? Is it consistent across different samples?
Moreover, it is not clear why the authors have limited the evaluation to the case where only two “chunks” are used. In principle, the method could be applied with many more subsets of latent variables and then manually inspect them to check it they are semantically meaningful (see [2]) 
*As previously mentioned, there are many recent works addressing the same problem from a fully-unsupervised perspective [1,2,3]. All these works provide quantitative results evaluating the learned representations by using them to predict real labels (e.g, attributes in the CelebA data-set). The authors could provide a similar evaluation for their method by using the feature representations learned by the siamese networks in order to evaluate how much information they convey about real factors of variation. This could clarify the advantages of the weakly-supervised strategy compared to unsupervised approaches.
Review summary
+The addressed problem (learning disentangled representations without explicit labeling) is challenging and interesting.
+The idea of using a proxy task (contrastive loss with triplets of generated images) is somewhat novel and promising.
- The authors report only negative results for the fully-unsupervised version of UD-GAN The paper lacks and in-depth discussion about why this negative result is interesting.
-The strategy proposed to provide weak-supervision to the model is too ad-hoc and it is not clear how to apply it in general applications
-The experimental section do not clarify the benefits of the proposed approach. In particular, the qualitative results are too limited and no quantitative evaluations is provided.
[1] Variational Inference of Disentangled Latent Concepts from Unlabelled Observations (Kumar et al, ICLR 2018)
[2] Beta-vae: Learning basic visual concepts with a constrained variational framework. (Higgins et. al, ICLR 2017)
[3] Disentangling Factors of Variation by Mixing Them. (Hu et. al, CVPR  2018)