Generating Videos with Scene Dynamics
We capitalize on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g. action classification) and video generation tasks (e.g. future prediction). We propose a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background. Experiments suggest this model can generate tiny videos up to a second at full frame rate better than simple baselines, and we show its utility at predicting plausible futures of static images. Moreover, experiments and visualizations show the model internally learns useful features for recognizing actions with minimal supervision, suggesting scene dynamics are a promising signal for representation learning. We believe generative video models can impact many applications in video understanding and simulation.
The paper describes a generative adversarial convolutional neural network for video (a block of 32 frames x 64x64 pixels). The architecture is divided into two streams one for static background and one for the moving foreground that are combined together using a mask that is also generated. The loss is a standard generative adversarial loss. The paper shows video generation experiments on four scene categories with qualitative results and quantitative user-study evaluating the realism of the generated videos. In addition, the paper uses trained representation on 5000 hours of Flickr videos for action classification demonstrating that the resulting representation serves as a better initialization for training  action classification models.
Strengths:  - The addressed problem of generative models of videos is interesting, timely and difficult. - Novelty: The proposed two stream architecture (static background / moving foreground + mask) for video generation is novel (albeit somewhat incremental over the previous static image generation GANs). Nevertheless, I like the proposed extension. - Experiments: The set of experiments is fairly comprehensive (generation, classification, user study). The results are encouraging, but but the visual quality of the generated results is quite poor and the action recognition results are much below the current state-of-the-art on the considered UCF dataset.  Weaknesses:  - The paper is somewhat incremental. The developed model is a fairly straighforward extension of the GAN for static images. - The generated videos have significant artifacts. Only some of the beach videos are kind of convincing.  The action recognition performance is much below the current state-of-the-art on the UCF dataset, which uses more complex (deeper, also processing optic flow) architectures.  Questions: - What is the size of the beach/golf course/train station/hospital datasets? - How do the video generation results from the network trained on 5000 hours of video look?  Summary: While somewhat incremental, the paper seems to have enough novelty for a poster. The visual results encouraging but with many artifacts. The action classification results demonstrate benefits of the learnt representation compared with random weights but are significantly below state-of-the-art results on the considered dataset.