Object-Oriented Dynamics Predictor
Generalization has been one of the major challenges for learning dynamics models in model-based reinforcement learning. However, previous work on actionconditioned dynamics prediction focuses on learning the pixel-level motion and thus does not generalize well to novel environments with different object layouts. In this paper, we present a novel object-oriented framework, called object-oriented dynamics predictor (OODP), which decomposes the environment into objects and predicts the dynamics of objects conditioned on both actions and object-to-object relations. It is an end-to-end neural network and can be trained in an unsupervised manner. To enable the generalization ability of dynamics learning, we design a novel CNN-based relation mechanism that is class-specific (rather than object-specific) and exploits the locality principle. Empirical results show that OODP significantly outperforms previous methods in terms of generalization over novel environments with various object layouts. OODP is able to learn from very few environments and accurately predict dynamics in a large number of unseen environments. In addition, OODP learns semantically and visually interpretable dynamics models.
This paper addresses the problem of action-conditional video prediction via a deep neural network whose architecture specifically aims to represent object positions, relationships, and interactions. The learned models are shown empirically to generalize to novel object configurations and to be robust to minor changes in object appearance.  Technical Quality  As far as I can tell the paper is technically sound. The experiments are well-designed to support the main claims. I especially appreciated the attempts to study whether the network is truly capturing object-based knowledge as a human might expect (rather than simply being a really fancy pixel -> pixel model).  Clarity  Honestly, this paper is a difficult read. A significant chunk of it is a description of a very complicated neural network architecture and I found it very hard to follow. That said, the architecture itself is one of the primary contributions and it must be described. Further, space limitations prevent a more relaxed walkthrough of the approach. Essentially, I'm afraid that I don't have any brilliant ideas about how to improve the reader's experience. I guess I would just say to the authors that it would be valuable for them to go over Section 3 with an eye toward giving high-level intuition and grounding the architectural choices in examples. If the authors can give the reader anything additional to help build this structure in their head and understand the intuitive function of each component, it will help.  Originality/Significance  The problem the paper is attempting to tackle is important and of broad interest. While certainly many of the components of the approach are familiar, the real trick is in composing them in a way that they will work together. The result is what seems to me to be a significant achievement and definitely a novel contribution. This approach can start with pixels, segment the scene into intuitive objects/object classes, and verifiably apply learned and generalizable object relations to predict movement. I'd say that's nothing short of remarkable; people are going to want to hear about this.  I will say, however, that I do wish the paper had spent a little bit of time discussing the weakness/limitations of the approach. What are some types of dynamics that it is not likely to handle well and what are the core challenges? For instance, based on my understanding, it seems that it would not be well-suited to problems where objects can appear and/or disappear (since it only predicts movement for each object already on the screen). It would just be nice to look up and survey what this approach addresses and what significant hurdles remain.  Overall  This paper represents a significant advance in the challenge problem of action-conditional video prediction. Though I wish it were easier to extract high-level intuition and big ideas from the description of the network architecture, I understand the challenge of clearly presenting such a complicated structure. The approach is well-evaluated, experiments validating that the model is able to exploit object-relational structure as intended.  ---After Author Response---  Having considered the other reviews and the author response I have decided to retain my original score. I frankly do not understand the arguments of the other reviewers, who seem to be dismissing the novel and important contribution of the paper (an end-to-end architecture that verifiably develops an object-oriented representation from raw perception) and demanding that the paper solve a bunch of other really hard, open problems (like dealing with variable numbers of objects, large numbers of object types, dynamic backgrounds, agent-centric positions, etc.). I continue to be impressed by the results and satisfied by the well-designed experiments that investigate the capabilities and behavior of the network. Though I frankly don't see where there's room, if the authors can squeeze the Mars Rover experiment into the paper as well, that would alleviate concern that this is a one-trick pony.  To the authors, I suppose I can only say that the kind of discussion that is present in the response about what the model can and cannot do and how one might imagine extending/building upon these ideas to handle more complicated/interesting problems is precisely what the paper needs. If this paper is rejected, which would be a shame in my estimation, I strongly encourage the authors to submit this paper again. Perhaps a robust discussion along these lines in the paper itself would help to highlight the significance of the contribution in the context of these bigger goals.