Unsupervised Learning for Physical Interaction through Video Prediction
A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment. Many existing methods for learning the dynamics of physical interactions require labeled object information. However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical. To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames. Because our model explicitly predicts motion, it is partially invariant to object appearance, enabling it to generalize to previously unseen objects. To explore video prediction for real-world interactive agents, we also introduce a dataset of 59,000 robot interactions involving pushing motions, including a test set with novel objects. In this dataset, accurate prediction of videos conditioned on the robot's future actions amounts to learning a "visual imagination" of different futures based on different courses of action. Our experiments show that our proposed method produces more accurate video predictions both quantitatively and qualitatively, when compared to prior methods.
The paper presents three deep learning methods for predicting future images of a scene. The authors learn a distribution of the pixels in the previous scene for each pixel. The first method predicts pixel movements and the other two are designed to implicitly estimate object movements. The network includes multiple masks including one for the background. The methods were evaluated on pushing data collected using multiple robots and the Human3.6M dataset. 
The paper mentions appendices and a link to videos, but the link does not work and the appendices are not included in the main file nor the supplementary material. I am reviewing the paper as it is.  The descriptions of the three methods need to be expanded. The method descriptions rely heavily on Fig. 1 and assume that most of the details are clear from the network structure. I did not find this to be the case. What is the structure of the m/M? Do the they adapt to the local image patches or are they constant over the entire image and rely on the masks? The authors should consider including a visualization of an m/M. Can they be interpreted?  The amount of data collected using the robot setup is impressive. The methods also outperform the state of the art. That being said, the paper is missing a discussion of the methods’ weaknesses and limitations. For example, why is the image resolution 64x64 pixels? Is it possible to use higher resolutions? The pushing task mainly involves planar movements parallel to the image plane. Can the method be applied to more 3D movements like a can tipping over? The authors should consider showing an example of a larger turning motion from the Human3.6M dataset for figure 7.  The qualitative results also leave room for improvement. There is a considerable amount of blurring in the images.  The task of predicting videos is obviously very challenging and I would not expect perfect results. However, the authors need to discuss the sources of these errors and how they may be addressed in the future.   The authors could consider exploiting the robot setup more for the evaluations. Can the robot grasp a moved object using only the predicted image? Is the quality of the prediction sufficient?  The MSE criterion is based on the optical flow computed for the ground truth data and the predicted data. It would be helpful to include an example image showing the optical flow computed for both images. How well does the optical flow handle the blurring?  Are there situations where the authors would recommend one of their proposed methods over the others?  The authors could merge some of the plots to make additional space for discussions.   How does the performance change when the background masks are removed?  Overall, the paper presents an interesting method for a very challenging task. The authors also present a large new dataset for video prediction. The network structure is impressive and seems well thought through. I therefore think that the authors have done great work. This work is unfortunately not reflected in the paper. Adding more detailed descriptions and thorough discussions of the methods’ limitations would make the paper more accessible and significantly increase the paper’s long term impact.