Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of rendered 3D scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.
The paper proposes a generative model for images. There's a probability mask per-pixel per-component (which yields mixing probabilities), and then a set of latents per-component that yield an image. The system is tested on a set of scenes like the GQN dataset, stacks of blocks, and the multi-dsprites dataset. The system is better than MONet, although there are a few lingering questions.
Summary of positives:
+ The factoring of the image into various components is eminently sensible and more work should build in the notion of objects
+ The method is well explained, and I found it easy to understand the entire process.
+ The method does appear to perform better than MONet, and qualitatively produces good results.
Summary of negatives:
- There are a few overclaims that should be fixed: namely that the system is a "generative model of 3D visual scenes" when in reality it is a generative model of images that has a latent space that is perhaps well-positioned to match images generated by 3D scenes composed of objects.
- The method section sets up a few questions that are never answered in the paper: GENESIS--S gets introduced as a baseline to test a hypothesis, but this never gets quantitatively evaluated (indeed GENESIS-S beats GENESIS in 2/3 of the categories where they are compared head-to-head); similarly, the fact that GENESIS produces probabilities is repeatedly sold as an advantage over other methods, but it's never used.
- The experiments are weak: the aforementioned questions aren't tackled, as well as a few other issues.
Overall, I lean ever so slightly towards rejection. The results are good looking, and the method seems well-explained. However, in my view, the manuscript seems to fall short of the mark. I am not, however, strongly opposed to the paper's acceptance. If it is accepted, I would urge authors to address the issues to make their paper have maximum impact.
Big picture things:
-I'm baffled by the repeated claim that this is a generative model of 3D scenes. In the introduction "first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes". I'm worried I'm just missing something profoundly obvious and critical -- since it seems obvious to me that it's not 3D. 
As far as I can tell, x here is a HxWxC image (where C is the number of channels and probably 3); the mixing probabilities are over HxW images, so it's a 2D segmentation system. It may be applied to photos rendered with a perspective camera -- but if the requirement for the system is being applied to images that come via projection, then is faster RCNN a 3D object detector? Are normalized cuts, SLIC superpixels, or any of the other myriad unsupervised segmentation methods from vision then 3D scene segmentation approaches?
-Similarly, it's not really a model for objects -- as can be seen in the multi-colored wall example in Fig 3, and Fig 8, 9. its notion of object is a region with uniform color (presumably due to the Gaussian p(x|z) ). So while it's great that it can model regions of uniform color, the system in large part seems to work (in its current form, reading the current manuscript) because the objects are all one color, and doesn't necessarily decompose the scene into objects.
Unless I'm totally missing something, this sort of claim that should be corrected very quickly. I realize that my personal bias is towards more engineering and harder data and I appreciate that there should be different operating points along the spectrum of how much information is provided. However, I think that it's worth making claims and assumptions clear, rather than claiming to solve the general problem on a specific case because of peculiarities about the specific case.
Method:
-The writing of the method is quite straightforward and written well. The system is clear to me. I largely have no complaints about the method. I do, however, have concerns about the experiments that are done to validate claims in the method section.
-GENESIS-S appears as this baseline "To investigate whether separate latents for masks and component appearances are necessary for decomposition", but then basically disappears. There are, as far as I can see, no qualitative results from it, and it basically appears once quantitatively, in S 4.3 / Table 1, where it appears to be as good if not better than GENESIS. In the appendix it's claimed that GENESIS trains faster than GENESIS-S, but if this is the selling point of GENESIS over GENESIS-S, it would be nice to have at least a little quantification of this.
-There is considerable fuss made over the fact that the system produces probabilities and this is repeatedly mentioned as an advantage over existing systems, but it is never demonstrated that these probabilities are good or useful. This leaves the reader hanging a bit. 
Experiments:
Overall the experiments are a little on the weak side.
+The qualitative results are good, but the primary selling point it seems is that the system is better than MONET. This does seem clear, and GENESIS does appear quite a bit better.
- While the results on factoring scenes do appear to be good, there's no quantitative evaluation of this (although the abstract promises it). Section 4.2 essentially says that ARI is bad, without demonstrating in a compelling way that it is. The paper says that this behavior can be seen in Appendix D, but this only shows the oversegmentation and not that this dramatically throws off performance -- I'd expect a figure showing a reversal of expectations -- a clearly worse result getting a clearly better ARI metric. 
Additionally, computer vision has been evaluating unsupervised segmentation for close to two decades (BSDS came out in 2001). The authors should look at the metrics in e.g., "Contour Detection andHierarchical Image Segmentation" Arbelaez et al. PAMI 2010. I realize that coming up with metrics is hard, but I think the burden is on the authors to find the metrics to show their conclusions quantitatively.
- The tower stability/height/view experiments seem incomplete. I find them quite interesting, but I'm not entirely sure of what to draw from it. The paper obliquely comments that the Groth paper gets better results with a more complex backbone network and blames it on lack of augmentation, use of a subset of the data, and a reduced resolution. But why not just do straightforward like train a MLP on an even further spatially downsampled image? Random here isn't a particularly compelling baseline or reference point. 
Small stuff that doesn't affect my review:
1) It might be worth explicitly pointing out that p_\theta(x_k,z_k^c) is just a Gaussian soon after Eqn 3. It's written after equation 5, but without making the p_\theta explicit. This may help some readers.
2) The tower stability is done with a one-hidden-layer (512) MLP. I'm of the personal opinion that readouts on latent variables are more informative if it's a linear transformation (since this can't do much heavy lifting).
3) The abstract promises semi-supervised learning -- is this the tower experiment? This strikes me more as transfer learning.
-----------------------------
Post review update: I have read the authors' responses, and found them thoughtful and to have answered my questions. I'm happy to accept the paper and would encourage the AC to do so.