We propose a generative latent variable model for unsupervised scene decomposition. Our model, SPACE, provides a unified probabilistic modeling framework to combine the best of previous models. SPACE can explicitly provide factorized object representation per foreground object while also decomposing background segments of complex morphology. Previous models are good at either of these, but not both. With the proposed parallel-spatial attention, SPACE resolves the scalability problem of previous methods and thus makes the model applicable to scenes with a much larger number of objects without performance degradation. Besides, the foreground/background distinction of SPACE is more effective and intuitive than other methods because unlike other methods SPACE can detect static objects that have been difficult to detect as foreground. In experiments on Atari and 3D-Rooms, we show that SPACE achieves the above properties consistently in all experiments in comparison to SPAIR, IODINE, and GENESIS.
Positives:
+The system makes sense and is explained well
+The factoring of scenes into objects and multiple background components is good
+I think overall the experiments are reasonable, although I have a number of questions about whether aspects of them are apples-to-apples
Negatives:
-Some of the experiments do not appear apples-to-apples
-There are a large number of changes, and there aren't any ablations. It's a little hard to follow and verify that the gains are credited properly. 
Overall, I'm favorably inclined towards accepting this paper so long as the experiments are more clearly made apples to apples. Right now, since I'm forced to give a binary decision and I'm not positive about comparisons, I have to lean towards rejection -- I'd peg my actual rating as 4.5.
Method: 
+The method is well-explained and straight-forward (in a good way). 
+The factoring of scenes into objects and multiple background components is good
+The parallelization is good, and the fact that it works far faster than SPAIR with similar results is quite nice
Experiments:
+Overall the experiments are pretty good and compare against the baselines I would expect, and have both qualitative and quantitative results.
+The method appears to do a good job of segmenting the objects, and if Figure 1 is representative, this is quite impressive. 
-Why does Figure 1 show results from different systems on different images? This makes comparison impossible. Paired samples are always more informative.
-It's not clear to me that fair comparisons were done, especially to GENESIS. 
(a) It's never listed how K for genesis was picked -- this should presumably be tuned somewhere to optimize performance. The paper mentions in the 4.2 that it was impossible to run the experiments for GENESIS for more than 256 components -- but the GENESIS paper has numbers more like K=9. If there are an overabundance of components, this might explain some of the object splitting observed in the paper.
(b) Unless I'm missing something, in Figure 5, for 4x4 and 8x8, it doesn't appear that IODOINE or GENESIS have converged at all. Does the validation MSE just then flatten (or go up) there? This is also wall-clock, so I'm not sure why things would stop there. This seems to conflate training speed with performance (although also note that the wall clock times being discussed are pretty small -- the rightmost side of the graph is ~14 hours -- hardly a burdensome experiment).
(c) Similarly, for 16x16 cells, SPAIR seems to be improving consistently. Is it being cut off?-Figure 5 -- The caption for the figure things appears to not make sense: GENESIS is listed as having K = HxW+5 components and SPACE has K=5 listed. Neither make sense to me. Are they out of order?
(d) For SPAIR in Table 1, it's not clear whether it's the slow SPAIR that was mentioned previously or the fast one (e.g., the predicted boxes are described as the same quality as SPAIR -- but is it the slow SPAIR or fast one?). I think the paper would benefit from being a bit clearer about this. I get that the parallel decomposition, in some sense, may be necessary to get any results. But I wish the paper were a bit more explicit.
-There are some reasonable results. I realize that there isn't existing ground truth on atari and other games, but why not label a few hundred frames manually? 
-It would have been nice to have an ablation of some of the components, including the boundary loss. Unfortunately, there's a complex multi-part system and it's not clear how to break off components apart for reuse elsewhere.
Small stuff that doesn't affect my review:
1) Figure 5 -- the figure text size is tiny and should be fixed. 
2) Eqn 3, subscript of the product "i" -> "i=1"
3) Table 1 -- captions on tables go on top
4) Now that the systems work like this, I'd encourage the authors to go and try stuff on more realistic data.
5) I would be a little wary of making a big deal out of the discovery of the Montezuma's Revenge key. I realize this is indeed important, but I don't see why something like the slic superpixel objective, or felzenswalb-huttenlocher wouldn't find it either. I think it's great that there's a setting in terms of network capacity (for fg/bg networks) that yields this result, but this seems to depend heavily on the particular networks used for each of the parts of the method, and not on the general method. Also, it seems largely a function of the fact that they're a small region with a different color.
-----------------------------------------
Post-rebuttal update: I have read the authors' response and I am happy to increase my rating to 6.