We consider the problem of learning models of intuitive physics from raw, unlabelled visual input. Differently from prior work, in addition to learning general physical principles, we are also interested in learning "on the fly" physical properties specific to new environments, based on a small number of environment-specific experiences. We do all this in an unsupervised manner, using a meta-learning formulation where the goal is to predict videos containing demonstrations of physical phenomena, such as objects moving and colliding with a complex background. We introduce the idea of summarizing past experiences in a very compact manner, in our case using dynamic images, and show that this can be used to solve the problem well and efficiently. Empirically, we show, via extensive experiments and ablation studies, that our model learns to perform physical predictions that generalize well in time and space, as well as to a variable number of interacting physical objects.
Summary 
The paper proposes an architecture for few-shot video prediction in which a number of videos are summarized through global pooling operations and passed into a video predictor that learns to leverage them for adaptation, similar in spirit to the RNN-based meta-learning approaches such as Santoro’16, Duan’16. Due to global image and feature pooling operations, the proposed approach is computationally efficient. Proof-of-concept experiments are presented in a simple simulated physical prediction setting. It is claimed that the proposed model achieves generalization to longer sequences and larger board sizes, as well as a larger number of objects.
Decision
The work considers an interesting task and shows promise, but the paper lacks a substantial contribution at this time. I recommend weak reject. To improve the paper, I recommend considering harder prediction tasks and lifting some of the restrictions of the method.
Pros
The paper proposes and evaluates a model for few-shot video prediction, which is an interesting and well-motivated task. 
Cons
1. The method is only evaluated on a toy task. While a model is proposed that can successfully solve the presented toy task, certain assumptions, such as averaging directly in the image space, may prevent the findings of this paper to scale to more complex tasks, which demands further investigation.
2. It is claimed that the method generalizes to longer sequences and larger board sizes, however, the performance of the method seems to quickly deteriorate in both settings. Authors present anecdotal evidence that the method generalizes to larger number of objects, however, quantitatively the results again are much inferior to the method trained on a larger number of objects. While the presented qualitative generalization findings are interesting, they seem limited in scope, especially when taking into account the simplicity of the considered data.
Minor comments
1. On page 3, the paper states “We are not the first to consider a similar learning problem, although most prior works do require some form of external supervision.” While there is a substantial body of literature that uses extra supervision, the following foundational works do not: Finn et al, 2016a,b; Fragkiadaki, 2016. In addition, “a similar learning problem” is considered without supervision in many papers on video prediction, including: Ranzato et al., 2014; Srivastava et al., 2015; Mathieu et al., 2015; Denton et al., 2017, 2018; Villegas et al., 2017, 2019; Wichers et al., 2018; Castrejon et al., 2019. While it might not be necessary to cite all of these as they do not necessarily apply their models to intuitive physics, the statement in the paper seems rather unsubstantiated.
2. A similar meta-learning prediction problem is addressed in Nagabandi’19a,b, which are not cited.
3. The paper is somewhat cluttered with notation, which overall hampers the flow for the reader. The paper would benefit from slight reorganization to improve the flow. This could be done e.g. be structuring the paper to first present the computational task that it addresses, namely few-shot video prediction, and how each component of the model works to address that problem, and only later go into the detail of how the data were collected and how each individual module was implemented
Santoro et al, Meta-learning with memory-augmented neural networks.
Duan et al, RL2: Fast reinforcement learning via slow reinforcement learning.
Nagabandi et al, LEARNING TO ADAPT IN DYNAMIC, REAL-WORLD ENVIRONMENTS THROUGH META-REINFORCEMENT LEARNING,
Nagabandi et al, DEEP ONLINE LEARNING VIA META-LEARNING: CONTINUAL ADAPTATION FOR MODEL-BASED RL
--------------------- Update 11.19 -----------------------
I appreciate the newly provided qualitative generalization results (in response to R3). However, it is clear that the network has trouble maintaining the exact number of spheres in the scene, especially after collisions or interactions with obstacles. While this generalization finding is undoubtedly interesting, I am not convinced it is enough for a publication in ICLR.
Furthermore, to support the authors' claim that the prediction performance on long sequences is satisfactory, I suggest comparison with a hand-crafted baseline, such as one suggested by R1, or one that makes use of explicit physics.