Stochastic Adversarial Video Prediction
Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging—the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior works in these aspects.
 This paper proposes to extend VAE-GAN from the static image generation setting to the video generation setting. It’s a well-written, simple paper that capitalizes on the trade-off between model realism and diversity, and the fact that VAEs and GANs (at least empirically) tend to lie on different sides of this spectrum. The idea to extend the use of VAE-GANs to the video prediction setting is a pretty natural one and not especially novel. However, the effort to implement it successfully is commendable and will, I think, serve as a good reference for future work on video prediction. There are also several interesting design choices that I think are worth of further exposition. Why, for example, did the authors only perform variational inference with the current and previous frames? Did conditioning on additional frames offer limited further improvement? Can the blurriness instead be attributable to the weak inference model? Please provide a response to these questions. If the authors have any ablation studies to back up their design choices, that would also be much appreciated, and will make this a more valuable paper for readers. I think Figure 5 is the most interesting figure in the paper. I would imagine that playing with the hyperparameters would allow one to traverse the trade-off between realism and diversity. I think having such a curve will help sell the paper as giving the practitioner the freedom to select their own preferred trade-off. I don’t understand the claim that “GANs prioritize matching joint distributions of pixels over per-pixel reconstruction” and its implication that VAEs do not prioritize joint distribution matching. VAEs prioritize matching joint distributions of pixels and latent space: min KL(q(z, x) || p(z, x)) and is a variational approximation of the problem min KL(q(x) || p(x)), where q(x) is the data distribution. The explanation provided by the authors is thus not sufficiently precise and I recommend the retraction of this claim. Pros: + Well-written + Natural extension of VAE-GANs to video prediction setting + Establishes a good baseline for future video prediction work Cons: - Limited novelty - Limited analysis of model/architecture design choices
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Stochastic Adversarial Video Prediction
Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging—the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior works in these aspects.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes to extend VAE-GAN from the static image generation setting to the video generation setting. It’s a well-written, simple paper that capitalizes on the trade-off between model realism and diversity, and the fact that VAEs and GANs (at least empirically) tend to lie on different sides of this spectrum. The idea to extend the use of VAE-GANs to the video prediction setting is a pretty natural one and not especially novel. However, the effort to implement it successfully is commendable and will, I think, serve as a good reference for future work on video prediction. There are also several interesting design choices that I think are worth of further exposition. Why, for example, did the authors only perform variational inference with the current and previous frames? Did conditioning on additional frames offer limited further improvement? Can the blurriness instead be attributable to the weak inference model? Please provide a response to these questions. If the authors have any ablation studies to back up their design choices, that would also be much appreciated, and will make this a more valuable paper for readers. I think Figure 5 is the most interesting figure in the paper. I would imagine that playing with the hyperparameters would allow one to traverse the trade-off between realism and diversity. I think having such a curve will help sell the paper as giving the practitioner the freedom to select their own preferred trade-off. I don’t understand the claim that “GANs prioritize matching joint distributions of pixels over per-pixel reconstruction” and its implication that VAEs do not prioritize joint distribution matching. VAEs prioritize matching joint distributions of pixels and latent space: min KL(q(z, x) || p(z, x)) and is a variational approximation of the problem min KL(q(x) || p(x)), where q(x) is the data distribution. The explanation provided by the authors is thus not sufficiently precise and I recommend the retraction of this claim. Pros: + Well-written + Natural extension of VAE-GANs to video prediction setting + Establishes a good baseline for future video prediction work Cons: - Limited novelty - Limited analysis of model/architecture design choices
assistant
 This paper proposes to extend VAE-GAN from the static image generation setting to the video generation setting. It’s a well-written, simple paper that capitalizes on the trade-off between model realism and diversity, and the fact that VAEs and GANs (at least empirically) tend to lie on different sides of this spectrum. The idea to extend the use of VAE-GANs to the video prediction setting is a pretty natural one and not especially novel. However, the effort to implement it successfully is commendable and will, I think, serve as a good reference for future work on video prediction. There are also several interesting design choices that I think are worth of further exposition. Why, for example, did the authors only perform variational inference with the current and previous frames? Did conditioning on additional frames offer limited further improvement? Can the blurriness instead be attributable to the weak inference model? Please provide a response to these questions. If the authors have any ablation studies to back up their design choices, that would also be much appreciated, and will make this a more valuable paper for readers. I think Figure 5 is the most interesting figure in the paper. I would imagine that playing with the hyperparameters would allow one to traverse the trade-off between realism and diversity. I think having such a curve will help sell the paper as giving the practitioner the freedom to select their own preferred trade-off. I don’t understand the claim that “GANs prioritize matching joint distributions of pixels over per-pixel reconstruction” and its implication that VAEs do not prioritize joint distribution matching. VAEs prioritize matching joint distributions of pixels and latent space: min KL(q(z, x) || p(z, x)) and is a variational approximation of the problem min KL(q(x) || p(x)), where q(x) is the data distribution. The explanation provided by the authors is thus not sufficiently precise and I recommend the retraction of this claim. Pros: + Well-written + Natural extension of VAE-GANs to video prediction setting + Establishes a good baseline for future video prediction work Cons: - Limited novelty - Limited analysis of model/architecture design choices