Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-world video frames. We also show that our model can be applied to visual analogy-making, and present an analysis of the learned network representations.
Paper proposes a model for synthesizing likely future frames form a single input image. The difference from the existing literature is that the paper proposes a probabilistic framework that allows sampling potentially a variety of future frames (from the conditional distribution). The model encodes images as feature maps and differential motion information as convolutional kernels. The two are then combined using proposed cross-convolution layer to produce final predictions. The conditional distribution of future frames is modeled using a conditional variational auto encoder and the predictions are made for the intensity changes (Eulerian motions) that are then combined with the original image to produce final predictions (as opposed to predicting raw pixels of the future image). The approach is illustrated on synthetic and more realistic datasets with promising results.
Overall the paper addresses an interesting problem and the formulation is also intriguing. However, lack of more through ablation studies makes it unclear which parts of the model are truly important. The baselines are also rather weak as only a very simple kNN approach is tested as the baseline. A more through evaluation would make the paper considerably stronger. Generally I actually liked the paper for its ideas and addressed problem, but would like to read authors rebuttal before finalizing the decision.  > References  Generally, references are adequate and well discussed. Activity forecasting is also related to the proposed problem, though, typically deals with individual agents and their trajectories as opposed to an image as a whole. I would suggest citing and discussing some of those works. A good overview is given in:  Social LSTM: Human Trajectory Prediction in Crowded Spaces A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, S. Savarese CVPR, 2016  > Assumption  - An intrinsic assumption of the work, it appears, that predictions are made under assumption of effectively static camera (this appears to be the reason why camera motion is factored out in the experiments as a pre-processing step). I do not view this as a negative, but do think that stating this assumption more clearly earlier on would be helpful for the reader.  - It is unclear from either motivation or the description why latent variables, z, in Figure 2 (d) are really necessary. The direct motion prediction (Figure 2(b)) could also be done presumably probabilistically, where given an image I a conditional distribution over the difference image could be modeled. What is the presumptive benefit of introducing latent variables z? I would like authors to address this in the rebuttal.   > Typos  - Page 3, bottom line, “frame without almost no motion” -> “frame with almost no motion”    > Experiments  Overall the experiments are interesting and compelling. However, the only baseline is rather weak (10 nearest neighbors to the input image with random sampling among the 10). In the very least a non-parametric generative model should be considered as a baseline. In particular, consider taking features of the image f(I) and a difference image and forming a KDE (e.g., Gaussian kernel on every sample) from joint samples to model p(I,v). This KDE can then be trivially conditioned (gaussian mixture conditioned is still a gaussian mixture) to obtain p(v|I). This conditional KDE can then be sampled or evaluated. This is still simple, but my guess is that it will work better then NN baseline and is also more inline with the premise of the paper.