Video Prediction via Selective Sampling
Most adversarial learning based video prediction methods suffer from image blur, since the commonly used adversarial and regression loss pair work rather in a competitive way than collaboration, yielding compromised blur effect. In the meantime, as often relying on a single-pass architecture, the predictor is inadequate to explicitly capture the forthcoming uncertainty. Our work involves two key insights: (1) Video prediction can be approached as a stochastic process: we sample a collection of proposals conforming to possible frame distribution at following time stamp, and one can select the final prediction from it. (2) De-coupling combined loss functions into dedicatedly designed sub-networks encourages them to work in a collaborative way. Combining above two insights we propose a two-stage framework called VPSS (Video Prediction via Selective Sampling). Specifically a Sampling module produces a collection of high quality proposals, facilitated by a multiple choice adversarial learning scheme, yielding diverse frame proposal set. Subsequently a Selection module selects high possibility candidates from proposals and combines them to produce final prediction. Extensive experiments on diverse challenging datasets demonstrate the effectiveness of proposed video prediction approach, i.e., yielding more diverse proposals and accurate prediction results.
The authors propose a method for video prediction with a particular focus on the problem of minimizing blur in predicted frames while optimizing for precision in motion forecasting. The main contribution is an introduced sampling module that explicitly separates different modes in the motion component and thus ensures disentangling the uncertainty in possible futures from visual quality of predicted frames.   It is an interesting and to the best of my knowledge original work, both in terms of the theoretical framework and the obtained results. The authors test their implementation on three commonly used benchmarks (MovingMNIST, RobotPush and Human3.6M) by predicting ~10 frames in the future and show significant visual improvements compared to the state-of-the-art, both in terms of visual quality and plausibility of movements.  Overall, it was an enjoyable read - however, the clarity of the manuscript can be improved, as at this point some parts are rather difficult to parse. It also contains a certain number of typos and minor grammar mistakes (some of them are listed below).  In general, the necessity of each of the proposed blocks could be more explicitly explained and justified by ablation studies. The "combiner" module, for example, should be better motivated: the principled difference between the top candidates and the final result is unclear. What is the meaning of masks in this case and how do they look in practice?  It would also be interesting to analyze the performance, possible failure modes and the nature of learned conditional distributions of the selector module - for now there is no discussion on this in the paper.  I'm not totally convinced that the experiment described in Table 1 is necessary - these results are rather intuitive and expected. It could make sense to sacrifice it in the interest of space and give more detailed description of the main contributions.  The experimental studies are relevant but not exhaustive. It looks like the authors do not provide any quantitative evaluation and comparison with the state-of-the-art on the mentioned Human3.6M dataset. For RobotPush, the only quantitative comparison is expressed in inception scores.  Figures 1-3 would be more helpful if the authors added notations used in the text.  There is something funny happening with notations for time stamps. L93-94 state the problem as "given x_t, predict x_{t+1}". However, eq. (1) contains x_t and x_{t-1} instead. The next paragraphs (L106-113) talk about all three of them: x_{t-1}, x_t and x_{t+1}. Eq. (5) should probably read k=min|x_{t+1}-\hat{x}_{t+1}| (otherwise, it points to a degenerate solution \hat{x}_t=x_t).  Assuming that in reality two frames x_{t-1} and x_t are given to predict the third frame x_t, one wouldn't expect the network to generalize beyond linear motion - is it indeed the case? In practice, does extending the temporal dimension of the input affect the accuracy of motion estimation in longer term?  The relationship between W, \Delta W and proposals X is not clear to me. Figure 2 hints that W are convolutional kernels of the sampling decoder (but it's a 3-layer ResNet, according to Section 3.3) but doesn't show how Ws are generated. There is an arrow saying "Element-wise addition" but is has a single input \Delta W_i. If active components of W are different for different proposals \hat{x}_t^i, what prevents us from learning them directly using eq. (5) by selectively updating sub-bands W_i?  Symbol W is overloaded and stands for both frame width and kernels.  The discriminator in Eq. (1) optimizes for "realism" of proposals x_t regardless of their "compatibility" with the past frames x_{t-1}. I realize this compatibility aspect is taken into account later by predicting "confidence scores", but I'm curious how conditional and unconditional discriminators would compare in this setting (if the authors have experimented with this).  L144: why exactly is the per-pixel L1 norm better applicable to linear motion?  Section 3.3. The authors mention curriculum learning "along the temporal dimension" but do not explain how it is implemented in their framework.   Table 2, third column: it's not clear what is meant by "inputs" and "outputs" (are the inputs processed / outputs predicted at once or in a sliding window manner? is it the same for all frameworks? how is it compatible with the description in Section 3?)  How do the authors explain that the performance on Human3.6M simultaneously goes down in terms of "reality" (23.1%) but improves in terms of "image quality" (40.8%)? (Table3, last column).  Section 4, Figure 4. Could the authors comment on why the proposed method is compared to DFN and SVG on Moving MNIST (in terms of PSNR and SSIM), but to SV2P and CDNA on RobotPush (in terms of inception scores)? It would be also interesting how performance of these methods compares to trivial baselines (copying the last observed frame, warping with optical flow).  L262-267. Which dataset do the author refer to? Please also mention it in the caption of Table 4. The performance is improved by increasing K from 1 to 2 - what happens when N and/or K grows further? Again, I would like to see more empirical evidence justifying the presence of each component: can we remove the selector (i.e. set K=N)? or the combiner?  Nitpicking: L30: is is possible -> it is possible L36: As show -> As shown L48: Nitish et al. -> Srivastava et al. L49: [32] propose -> [32] proposes L54: methods suffers -> method suffer L61: inference is not a verb L76: 34 twice L117: Rivera et al. -> Guzman-Rivera et al. L228: As show -> as shown Ref [23]: CoRR -> ICLR, 2014 Ref [29]: CoRR -> ICLR, 2016 etc.