Can we build models that automatically learn about object motion from raw, unlabeled videos? In this paper, we study the problem of multi-step video prediction, where the goal is to predict a sequence of future frames conditioned on a short context. We focus specifically on two aspects of video prediction: accurately modeling object motion, and producing naturalistic image predictions. Our model is based on a flow-based generator network with a discriminator used to improve prediction quality. The implicit flow in the generator can be examined to determine its accuracy, and the predicted images can be evaluated for image quality. We argue that these two metrics are critical for understanding whether the model has effectively learned object motion, and propose a novel evaluation benchmark based on ground truth object flow. Our network achieves state-of-the-art results in terms of both the realism of the predicted images, as determined by human judges, and the accuracy of the predicted flow. Videos and full results can be viewed on the supplementary website: https://sites.google.com/site/omvideoprediction.
This is a fine paper that generally reads as a new episode in a series on motion-based video prediction with an eye towards robotic manipulation [Finn et al. 2016, Finn and Levine 2017, Ebert et al. 2017]. The work is rather incremental but is competently executed. It is in line with current trends in the research community and is a good fit for ICLR. The paper is well-written, reasonably scholarly, and contains stimulating insights.
I recommend acceptance, despite some reservations. My chief criticism is a matter of research style: instead of this deluge of barely distinguishable least-publishable-unit papers on the same topic, in every single conference, I wish the authors didn’t slice so thinly, devoted more time to each paper, and served up a more substantial dish.
Some more detailed comments:
- The argument for evaluating visual realism never quite gels and is not convincing. The paper advocates two primary metrics: accuracy of the predicted motion and perceptual realism of the synthesized images. The argument for motion accuracy is clear and is clearly stated: it’s the measure that is actually tied to the intended application, which is using action-conditional motion prediction for control. A corresponding argument for perceptual realism is missing. Indeed, a skeptical reviewer may suspect that the authors needed to add perceptual realism to the evaluation because that’s the only thing that justifies the adversarial loss. The adversarial loss is presented as the central conceptual contribution of the paper, but doesn’t actually make a difference in terms of task-relevant metrics. A skeptical perspective on the paper is that the adversarial loss just makes the images look prettier but makes no difference in terms of task performance (control). This is an informative negative result. It's not how the paper is written, though.
- The “no adversary”/“no adv” condition in Table 1 and Figure 4 is misleading. It’s not properly controlled. It is not the case that the adversarial loss was simply removed. The regression loss was also changed from l_1 to l_2. This is not right. The motivation for this control is to evaluate the impact of the adversarial loss, which is presented as the key conceptual contribution of the paper. It should be a proper control. The other loss should remain what it is in the full “Ours” condition (i.e., l_1).
- The last sentence in the caption of Table 1 -- “Slight improvement in motion is observed by training with an adversary as well” -- should be removed. The improvement is in the noise.
- Generally, the quantitative impact of the adversarial loss never comes together. The only statistically significant improvement is on perceptual image realism. The relevance of perceptual image realism to the intended task (control) is not substantiated, as discussed earlier.
- In the perceptual evaluation procedure, the “1 second” restriction is artificial and makes the evaluated methods appear better than they are. If we are serious about evaluating image realism and working towards passing the visual Turing test, we should report results without an artificial time limit. They won’t look as flattering, but will properly report our progress on this journey. If desired, the results of timed comparisons can also be reported, but reporting just a timed comparison with an artificial limit of 1 second may mislead some readers into thinking that we are farther along than we actually are.
There are some broken sentences that mar an otherwise well-written paper:
- End of Section 1, “producing use a learned discriminator and show improvements in visual quality”
- Beginning of Section 3, “We first present the our overall network architecture”
- page 4, “to choose to copy pixels from the previous frame, used transformed versions of the previous frame”
- page 4, “convolving in the input image with”
- page 5, “is know to produce”
- page 5, “an additional indicating”
- page 5, “Adam Kingma & Ba (2015)” (use the other cite command)
- page 5, “we observes”
- page 5, “smaller batch sizes degrades”
- page 5, “larger batch sizes provides”