In this work, we propose a composition/decomposition framework for adversarially training generative models on composed data -data where each sample can be thought of as being constructed from a fixed number of components. In our framework, samples are generated by sampling components from component generators and feeding these components to a composition function which combines them into a "composed sample". This compositional training approach improves the modularity, extensibility and interpretability of Generative Adversarial Networks (GANs) -providing a principled way to incrementally construct complex models out of simpler component models, and allowing for explicit "division of responsibility" between these components. Using this framework, we define a family of learning tasks and evaluate their feasibility on two datasets in two different data modalities (image and text). Lastly, we derive sufficient conditions such that these compositional generative models are identifiable. Our work provides a principled approach to building on pre-trained generative models or for exploiting the compositional nature of data distributions to train extensible and interpretable models.
The paper proposes a framework for training generative models that work on composed data. The models are trained in an adversarial fashion. The authors apply it to decompose foreground/background parts on MNIST images, and to perform sentence composition/decomposition.
High level comments:
* Clarity: In terms of language and writing style, the paper is written very clearly and easy to follow. In terms of presentation, there are some details that are omitted which would have made understanding easier and the work more reproducible.
* Quality: The idea that is introduced seems intuitive and reasonable, but the experiments does not have enough details to prove that this method works (i.e. no quantitative results presented).  Moreover, the presentation of the method is not very well done (missing details), especially since the authors used the upper limit of 10 pages.
* Originality: I am not familiar with the literature of generative models to judge this precisely, but according to the related work section it sounds like an original idea that is worth sharing.
* Significance: I believe the idea of modeling data composition explicitly sounds intuitive and interesting, and it is worth sharing. However, the experimental section does not have enough evidence that it is actually possible to learn this, so it is not clear whether the contribution is significant.
Pros:
-	interesting new problem formulation 
-	simple and clear language
-	the theoretical analysis in the last section could be interesting more generally in the context of GANs
-	the framework is applied on 2 different modalities: images and text.
Cons:
-	hard to tell whether this approach works since the metrics for evaluation are not specified and there are no quantitative results in the experimental section (only 1 qualitative example per task)
-	the work is not reproducible due to the lack of details (see more explanations below)
-	the theoretical analysis is a standalone piece of the paper, without any discussion about the implications, or making connections to the previous sections.
Detailed comments:
1.	I believe the weakest part of this paper is the evaluation section. The authors run their framework on 4 tasks of increasing difficulty. While the MNIST examples make for a nice and intuitive qualitative analysis, the are no quantitative results at all. The only result that is reported for each task is one qualitative picture. The authors make statements such as “The decomposition network learns to decompose the digits and backgrounds correctly” , “Given one component, decomposition function and the other component can be learned.” but there is not mention for how these conclusion are made (no metrics, no numbers). Indeed, it is difficult in general to quantify the results of generative models, but most other GAN papers introduce some sort metric that can be used to aggregate the evaluation on an entire dataset. If the authors manually inspected the results, they should at least report how many images they inspected and how many looked correct. 
2.	Aside from evaluation, there are some other details missing from the presentation. The individual details may not be major, but because all of these are missing together, it really affects the overall quality of the paper. For example:
    	 the authors state: “To train discriminator(s), a regularization is applied. For brevity, we do not show the regularization term (see Petzka et al. (2017)) used in our experiments.”. For reproducibility purposes, I believe it is important to at least mention the type of regularization, at least in the appendix. 
    	There is a parameter alpha used to balance the losses. What values was used in the experiments?
    	Choices of models are often not explained. Why did you choose that form for c(o1, o2) in section 3.3? Why DCGAN for component generators, and U-net for decomposition?
    	It is not explained in detail how the Yelp-reviews dataset is altered to achieve coherence. The authors mention that “As we sample a pair independently, the input sentences are not generally coherent but the coherence can be achieved with a small number of changes.”. However, the specific algorithm by which these changes are made is not specified, and thus it can’t be reproduced.
3.	The theoretical section is an interesting contribution, but the paper just states a list of theorems without making any connections to the applications used before, or a broader discussion about how these fit in the context of GANs more generally.
4.	My understanding is that both datasets used are created by the authors by making alterations to MNIST and Yelp-reviews dataset, thus making them to some extent synthetic datasets suited to fit this problem formulation. I would have like to see how this composition/decomposition works on existing datasets with no alterations. Does it still work? 
5.	In section 2.3, in the coherent sentence experimental setting, I don’t fully understand the design of the task. Figure 2 shows an example where composition and decomposition are not symmetric (i.e. composing then decomposing does not go back to the input sentences), although one of your losses is supposed to ensure exactly this cyclic consistency. Why not choose another problem that doesn’t directly violate your assumptions?
Minor issues: 
6.	From the related work section, it is not clear how your approach is different from Azadi et al. (2018). Please include more details.
7.	In section 2.4, you mention using Wasserstein GANs, with no further details about this model (not even a one line description). Without reading their paper, the readers of your paper could not easily follow through this section. The losses further introduced are also not explained intuitively (e.g. what do the two expectation terms in l_g_i represent?).
8.	I believe there are some errors in which tasks reference which figures in section 3.3. Should Task 2 refers to Figure 6, and Task 3 to Figure 7?
9.	What exactly is range(.) in section 4? If this refers to the interval of values that a variable can take, the saying “is a matrix of size |range(Z)| × |range(Y )|” doesn’t exactly make sense. Please define formally. 
Final remarks and advice: 
Overall, I believe the paper introduces some interesting ideas. There is definitely value in the problem definition and theoretical analysis. However, I believe the paper needs more work on presentation and evaluation, especially since the authors opted for 10 pages and according to ICLR guidelines “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages.”. Hopefully the above comments will help the authors improve this work!