Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.
This paper presents a method for predicting future frames of a video (or unseen views of a 3D scene) in a "jumpy" way (you can query arbitrary viewpoints or timesteps) and "consistent" way (when you sample different views, the scene will be consistent). They use a VAE that encodes the input video in a permutation invariant way, which is achieved by summing the per-frame latent vectors. Then, they sample a latent vector using a DRAW prior. This latent vector can then be used to render the video/scene from different times/viewpoints via an LSTM decoder. They test the model on several toy datasets: they compare to video prediction methods on a dataset of moving shapes, and 3D viewpoint prediction on a 3D MNIST "dice" dataset.
Pros:
- The idea of developing new methods for viewpoint and video synthesis that allow for "jumpy" and "consistent" predictions is an important problem.
- The paper is fairly well written.
- The design of the model is reasonable (it is a natural extension of VAE viewpoint/future prediction methods).
Cons:
- All of the experiments were done on toy datasets. These are also not well-established toy datasets, and seem tailored to debugging the model, so it is not particularly surprising that the method worked. Since the main contribution is not very novel from a technical perspective (it is more about designing a model from existing, well-established components), this is a significant limitation. 
- The paper suggests with experiments that GQN generates predictions that are less consistent across samples, but it is not clear exactly which design decisions lead to this difference. Why is this model more jumpy and consistent than GQN?
- The paper claims that JUMP trains more reliably than several video prediction methods in Figure 5. Yet, in the 3D viewpoint synthesis task, they suggest that JUMP had trouble with convergence, i.e.: "We ran 7 runs for each model, and picked the best 6/7 runs for each model (1 run for JUMP failed to converge)." This is confusing for two reasons. First, why was this evaluation protocol chosen (i.e. running 7 times and picking the 6 best)? If it was a post-hoc decision to remove one training run, then this should be clarified, and the experiment should be redesigned and rerun. Second, is the implication that JUMP is more stable than video prediction methods, but not necessarily more stable than GQN for viewpoint prediction?
- The paper should consider citing older representation learning work that deals with synthesizing images from multiple viewpoints. For example: 
M. Tatarchenko, A. Dosovitskiy, T. Brox. "Multi-view 3D Models from Single Images with a Convolutional Network". ECCV 2016.
- There is insufficient explanation of the BADJ baseline. What architectural changes are different?
- The decision to use DRAW, instead of a normal VAE prior, is unusual and not explained in much detail. Why does this improve the visual fidelity of the samples?
Overall:
The paper does not present enough evidence that this model is better at jumpy/consistent predictions than other approaches. It is evaluated only on toy datasets: if the technical approach were more novel (and if it was clearer where the performance gains are coming from) then this could be OK, but it seems to be a fairly straightforward extension of existing models.