Object-based approaches for learning action-conditioned dynamics has demonstrated promise for generalization and interpretability. However, existing approaches suffer from structural limitations and optimization difficulties for common environments with multiple dynamic objects. In this paper, we present a novel self-supervised learning framework, called Multi-level Abstraction Objectoriented Predictor (MAOP), for learning object-based dynamics models from raw visual observations. MAOP employs a three-level learning architecture that enables efficient dynamics learning for complex environments with a dynamic background. We also design a spatial-temporal relational reasoning mechanism to support instance-level dynamics learning and handle partial observability. Empirical results show that MAOP significantly outperforms previous methods in terms of sample efficiency and generalization over novel environments that have multiple controllable and uncontrollable dynamic objects and different static object layouts. In addition, MAOP learns semantically and visually interpretable disentangled representations.
In this paper, the novel MAOP model is described for self-supervised learning on past video game frames to predict future frames. The presented results indicate that the method is capable of discovering semantically important visual components, and their relation and dynamics, in frames from arcade-style video games. Key to the approach is a multi-level self-learning approach:  more abstract stages focus on simpler problems that are easier to learn, which in turn guide the learning process at the more complex stages.
A downside is that it the method is complex, consisting of many specific sub-components and algorithms, which in turn have again other sub-components. This makes the paper a long read with a lot of repetition, and various times the paper refers to the names of sub-components that are only explained later. Other methodological details that are relevant to understand how the method operates are described in the Appendices. I expect that if the paper would be better structured, it would be easier to understanding how all the parts fit together. Another downside of this complexity is that the method seems designed for particular types of video game frames, with static backgrounds, a fixed set of objects or agents. It is unclear how the method would perform on other types of games, or on real-world videos. While the method therefore avoids the need for manual annotation, it instead encodes a lot of domain knowledge in its design and components.
I also didn't fully understand how the self-supervised model is used for Reinforcement Learning in the experiments. Is the MAOP first trained, and the fixed to perform RL with the learned agent models, or is the MOAP learned end-to-end during RL?
Pros:
+ MAOP seems successful on the tested games in the experiments
+ Demonstrates that, with a sufficiently engineered method, self-supervised learning can be used to discover different types of objects, and their dynamics.
Cons:
- writing could be improved, as the methodology currently reads as a summation of facts, and some parts are written out of order, resulting in various forward references to components that only become clear later. Several times, the paper states that some novel algorithm is used, but then provides no further explanation in the text as all description of this novelty is deferred to an appendix. 
- method does not seem generic, hence it is unclear how relevant this architecture it is to other use cases
- many hyperparameters for the individual components, algorithms. Unclear how these parameter setting affect the results
Below are more detailed comments and questions:
General comments:
* The proposed MOAP method consists of many subalgorithms, resulting in various (hyper)parameters which may impact the results (e.g. see Appendix A, B). Appendix D lists several used hyperparameter settings, though various parameters for the algorithms are still missing (e.g. thresholds alpha, beta in Algo.2). Were the used parameters optimized? How are these hyperparameters set in practice? How does changing them impact your results?
* Methods seems particularly designed for 'video games', where the object and background structures have well defined sizes, appearance, etc. How will the MOAP fair in more realistic situations with noisy observations, occluded objects, changing appearances and lighting conditions, etc.?
* How about changing appearance of an agent during an action, e.g. a 'walking animation' ? Can your method learn the sequence of sprites to accurately predict the next image? Is that even part of the objective?
* Appendix D has important implementation details, but is never mentioned in the text I believe! Didn't realize it existed on first read through.
* Introduction:
	* What prediction horizon are you targeting? 1 step, T steps into the future, 1 to T steps in the future simultaneously?
	What are you trying to predict? Object motion? Future observations?
	* "... which includes a CNN-based Relation Net to ... ", the names Relation Net, Inertia Net, etc.. are used as if the reader is expected to know what these are already. If these networks were introduced in related work already, please add citations. Otherwise please rephrase to clarify that these are networks themselves are part of your novel design.
* Section 3.1
	* "It takes multiple-frame video images ... and produce the predictions of raw visual observations.". As I understand from this, the self-supervised approach basically performs supervised learning to predict a future frame (target output) given past frames (input). I do not understand how this relates to Reinforcement Learning (RL) as mentioned in the introduction and Related Work. Is there still some reward function in play when learning the MAOP parameters? Or is the idea to first self-supervised learn the MAOP, and afterwards fix its parameters and use it in separate a RL framework? I believe RL is not mentioned anymore until Section 4.2. This connection between self-supervised and reinforcement learning should be clarified, or otherwise the related work should be adjusted to include other (self-supervised) work on predicting future image frames.
	* "An object mask describes the spatial distribution of an object ..." Does the distribution capture uncertainty on the object's location, or does it capture the spread of the object's extent ('mass distribution') ?
	* "Note that Object Detector uses the same CNN architecture with OODP". What does OODP stand for? Add citation here. (first mention of OODP is in Experiments section)
	* "(similar with Section 3.2)" → "similar to". Also, I find it a confusing to say something is similar to what will be done in a future section, which has not yet been introduced. Can you not explain the procedure here, and in Section 3.2 say that the procedure is "similar to Section 3.1" instead?
	* "to show the detailed structure of the Effect Net module." First time I see the name 'Effect Net', what is it? This whole paragraph different nets are named, with a rough indication of their relation, such as "Dynamic Net", "Relation Net" and "Inertia Net". Is "Effect Net" a different name for any of the three previous nets? The paper requires the reader to puzzle from Fig.2 that Relation Net and Inertia Net are parts of Effect Net, which in turn is part of Dynamics Net. This wasn't clear from the text at all.
* Section 3.2:
	* p7.: "Since DISN leans" → "Since DISN learns" ?
	* There are many losses throughout the paper, but I only see at the end of Section 3.1 some mentioning that multiple losses are combined. How is this done for the other components, .e.g is the total loss for DISN a weighted sum of L_foreground and L_instance ? Are the losses for all three three MAOP levels weighted for full end-to-end learning?
	* This section states various times "we propose a novel [method]", for which then no explanation is given, and all details are explained in the Appendix. While the Appendix can hold important implementation details, I would still expect that novelties of the paper are clearly explained in the paper itself. As it stands, the appendix is used as an extension of the methodological section of an already lengthy paper.
	* "Conversely, the inverse function is ... " M has a mask for each of the n_o "object classes", hence the "Instance Localization Module" earlier to split out instances from the class masks. So how can there be a single motion vector STN^-1(M,M') if there are multiple instances for an object mask? How will STN^-1 deal with different amount of instances in M and M' ?
* Section 3.3:
	* What is the output of this level? I expect some mathematical formulation as in the previous sections, resulting in some symbol, that is then used in Section 3.2. E.g. is the output "foreground masks F" (found in Appendix A) ?  This paper is a bit of a puzzle through the pages for the reader.
* Section 4: 
	* "We compare MAOP with state-of-the-art action-conditioned dynamics learning baselines, ..." Please re-iterate how these methods differ in assumptions, what they model, with respect to your novel method? For instance, is the main difference your "novel region proposal method" and such? Is the overall architecture different? E.g. explain here already the AC Model uses "pixel-level inference", and that OODP has "lacks knowledge on object-to-object relations" to underline their difference to your approach, and provide context for your conclusions in Section 4.1.
* Appendix A:
	* Algorithm 1, line 7: "sample a pixel coordinate" → is this non-deterministically sampling?