summaries: The paper studies conditional video prediction. In particular, it focuses on the problem of current models underfitting and not scaling to datasets. The paper proposes an architecture that is claimed capable of using parameters more efficiently in order to overfit. Then, data augmentation is introduced to improve performance for generalization. They evaluated their method against baselines on 4 datasets (Human 3.6M, KITTI, Robonet, BAIR pushing dataset). They also showed experiments to support overfitting/underfitting claims.   The paper proposes a simple and scalable variational video prediction model FitVid, which attains a better fit to video prediction datasets even with a similar parameter count as prior models. The author has observed that previous methods suffer from underfitting on these datasets, directly applying FitVid actually results in overfitting. The FitVid uses a set of existing image augmentation techniques to prevent overfitting so that it can achieve state-of-the-art results on several prediction benchmarks. FitVid's architecture is based on SE-UNet LSTM, which seems to be the common backbone for the stochastic video prediction task.  This paper presents a new network architecture called FitVid to perform the task of video prediction, i.e. the task of predicting future frames from previous frames. Previous methods for this tasks usually suffer from "underfitting", while FitVid is able to overfit on major benchmarks without increasing the number of parameters, thanks to a better model architecture. What is more, FitVid is much easier to train than previous work, without using any bells and whistles in training. As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics. This paper discusses a new framework named FITVID to handle the problem of video prediction. FitVid is built on existing modules like Sikp connection, Sequeeze and Excite, and LSTMs. FitVid only needs a simple training strategy and begins overfitting the datasets. Data augmentation is also adopted to handle the problem of overfitting. 
abstractText: An agent that is capable of predicting what happens next can perform a variety of tasks through planning with no additional training. Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for many other visual perception tasks. This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances. Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain. There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions. In this paper, we argue that the inefficient use of parameters in the current video models is the main reason for underfitting. Therefore, we introduce a new architecture, named FitVid, which is capable of fitting the common benchmarks so well that it begins to suffer from overfitting – while having similar parameter count as the current state-of-theart models. We analyze the consequences of overfitting, illustrating how it can produce unexpected outcomes such as generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques. As a result, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.
references.authors: Mohammad Babaeizadeh Chelsea Finn Dumitru Erhan Roy H Campbell Sergey Levine
references.title: Stochastic variational video prediction
references.authors: Mohammad Babaeizadeh Mohammad Taghi Saffar Danijar Hafner Harini Kannan Chelsea Finn Sergey Levine Dumitru Erhan
references.title: Models, pixels, and rewards: Evaluating design trade-offs in visual model-based reinforcement learning
references.authors: Samy Bengio Oriol Vinyals Navdeep Jaitly Noam Shazeer
references.title: Scheduled sampling for sequence prediction with recurrent neural networks
references.venue: arXiv preprint arXiv:1506.03099,
references.authors: Byron Boots Arunkumar Byravan Dieter Fox
references.title: Learning predictive models of a depth camera & manipulator from raw execution traces
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: James Bradbury Roy Frostig Peter Hawkins Matthew James Johnson Chris Leary Dougal Maclaurin George Necula Adam Paszke Jake VanderPlas Skye Wanderman-Milne Qiao Zhang
references.title: JAX: composable transformations of Python+NumPy programs, 2018
references.venue: URL http://github.com/google/jax
references.authors: Andreja Bubic D Yves Von Cramon Ricarda I Schubotz
references.title: Prediction, cognition and the brain
references.venue: Frontiers in human neuroscience,
references.authors: Arunkumar Byravan Dieter Fox
references.title: Se3-nets: Learning rigid body motion using deep neural networks
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Lluis Castrejon Nicolas Ballas Aaron Courville
references.title: Improved conditional vrnns for video prediction
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Alberto Cenzato Alberto Testolin Marco Zorzi
references.title: On the difficulty of learning and predicting the long-term dynamics of bouncing objects
references.authors: Baoyang Chen Wenmin Wang Jinzhuo Wang
references.title: Video imagination from a single image with transformation generation
references.venue: In Proceedings of the on Thematic Workshops of ACM Multimedia 2017,
references.authors: Kurtland Chua Roberto Calandra Rowan McAllister Sergey Levine
references.title: Deep reinforcement learning in a handful of trials using probabilistic dynamics models
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Aidan Clark Jeff Donahue Karen Simonyan
references.title: Adversarial video generation on complex datasets
references.venue: arXiv preprint arXiv:1907.06571,
references.authors: Ekin D Cubuk Barret Zoph Jonathon Shlens Quoc V Le
references.title: Randaugment: Practical automated data augmentation with a reduced search space
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,
references.authors: Sudeep Dasari Frederik Ebert Stephen Tian Suraj Nair Bernadette Bucher Karl Schmeckpeper Siddharth Singh Sergey Levine Chelsea Finn
references.title: Robonet: Large-scale multi-robot learning
references.venue: CoRL,
references.authors: Emily Denton Rob Fergus
references.title: Stochastic video generation with a learned prior
references.venue: arXiv preprint arXiv:1802.07687,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: arXiv preprint arXiv:1810.04805,
references.authors: Laurent Dinh David Krueger Yoshua Bengio
references.title: Nice: Non-linear independent components estimation
references.venue: arXiv preprint arXiv:1410.8516,
references.authors: Laurent Dinh Jascha Sohl-Dickstein Samy Bengio
references.title: Density estimation using real nvp
references.venue: arXiv preprint arXiv:1605.08803,
references.authors: Michael Dorkenwald Timo Milbich Andreas Blattmann Robin Rombach Konstantinos G Derpanis Bjorn Ommer
references.title: Stochastic image-to-video synthesis using cinns
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Lingyu Duan Jiaying Liu Wenhan Yang Tiejun Huang Wen Gao
references.title: Video coding for machines: A paradigm of collaborative compression and intelligent analytics
references.venue: IEEE Transactions on Image Processing,
references.authors: Frederik Ebert Chelsea Finn Alex X. Lee Sergey Levine
references.title: Self-Supervised Visual Planning with Temporal Skip Connections
references.venue: Conference on Robot Learning (CoRL),
references.authors: Frederik Ebert Chelsea Finn Alex X Lee Sergey Levine
references.title: Self-supervised visual planning with temporal skip connections
references.venue: In Conference on Robot Learning (CoRL),
references.authors: Frederik Ebert Sudeep Dasari Alex X Lee Sergey Levine Chelsea Finn
references.title: Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning
references.venue: In Conference on Robot Learning (CoRL),
references.authors: Frederik Ebert Chelsea Finn Sudeep Dasari Annie Xie Alex Lee Sergey Levine
references.title: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control
references.venue: arXiv preprint arXiv:1812.00568,
references.authors: Stefan Elfwing Eiji Uchibe Kenji Doya
references.title: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
references.venue: Neural Networks,
references.authors: Yuki Endo Yoshihiro Kanamori Shigeru Kuriyama
references.title: Animating landscape: self-supervised learning of decoupled motion and appearance for single-image video synthesis
references.authors: Chelsea Finn Sergey Levine
references.title: Deep visual foresight for planning robot motion
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Chelsea Finn Ian Goodfellow Sergey Levine
references.title: Unsupervised learning for physical interaction through video prediction
references.venue: In Advances in neural information processing systems,
references.authors: Chelsea Finn Xin Yu Tan Yan Duan Trevor Darrell Sergey Levine Pieter Abbeel
references.title: Deep spatial autoencoders for visuomotor learning
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Jean-Yves Franceschi Edouard Delasalles Mickaël Chen Sylvain Lamprier Patrick Gallinari
references.title: Stochastic latent residual video prediction
references.venue: arXiv preprint arXiv:2002.09219,
references.authors: Andreas Geiger Philip Lenz Christoph Stiller Raquel Urtasun
references.title: Vision meets robotics: The kitti dataset
references.venue: International Journal of Robotics Research (IJRR),
references.authors: Andreas Geiger Philip Lenz Christoph Stiller Raquel Urtasun
references.title: Vision meets robotics: The kitti dataset
references.venue: The International Journal of Robotics Research,
references.authors: Ian J Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville Yoshua Bengio
references.title: Generative adversarial networks
references.venue: arXiv preprint arXiv:1406.2661,
references.authors: Vincent Le Guen Nicolas Thome
references.title: Disentangling physical dynamics from unknown factors for unsupervised video prediction
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Danijar Hafner Timothy Lillicrap Ian Fischer Ruben Villegas David Ha Honglak Lee James Davidson
references.title: Learning latent dynamics for planning from pixels
references.venue: arXiv preprint arXiv:1811.04551,
references.authors: Danijar Hafner Timothy Lillicrap Jimmy Ba Mohammad Norouzi
references.title: Dream to control: Learning behaviors by latent imagination
references.venue: arXiv preprint arXiv:1912.01603,
references.authors: Jonathan Heek Anselm Levskaya Avital Oliver Marvin Ritter Bertrand Rondepierre Andreas Steiner Marc van Zee
references.title: Flax: A neural network library and ecosystem for JAX
references.authors: Dan Hendrycks Kevin Gimpel
references.title: Gaussian error linear units (gelus)
references.venue: arXiv preprint arXiv:1606.08415,
references.authors: Irina Higgins Loic Matthey Xavier Glorot Arka Pal Benigno Uria Charles Blundell Shakir Mohamed Alexander Lerchner
references.title: Early visual concept learning with unsupervised deep learning
references.venue: arXiv preprint arXiv:1606.05579,
references.authors: Irina Higgins Loic Matthey Arka Pal Christopher Burgess Xavier Glorot Matthew Botvinick Shakir Mohamed Alexander Lerchner
references.title: beta-vae: Learning basic visual concepts with a constrained variational framework
references.venue: International Conference on Learning Representations (ICLR),
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long short-term memory
references.venue: Neural computation,
references.authors: Kibeom Hong Youngjung Uh Hyeran Byun
references.title: Arrowgan: Learning to generate videos by learning arrow of time
references.venue: arXiv preprint arXiv:2101.03710,
references.authors: Jie Hu Li Shen Gang Sun
references.title: Squeeze-and-excitation networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Qiyang Hu Adrian Wälchli Tiziano Portenier Matthias Zwicker Paolo Favaro
references.title: Learning to take directions one step at a time
references.venue: In 2020 25th International Conference on Pattern Recognition (ICPR),
references.authors: Quan Huynh-Thu Mohammed Ghanbari
references.title: Scope of validity of psnr in image/video quality assessment
references.venue: Electronics letters,
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: In International conference on machine learning,
references.authors: Catalin Ionescu Dragos Papava Vlad Olaru Cristian Sminchisescu
references.title: Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments
references.venue: IEEE transactions on pattern analysis and machine intelligence,
references.authors: Xu Jia Bert De Brabandere Tinne Tuytelaars Luc V Gool
references.title: Dynamic filter networks
references.venue: In Advances in neural information processing systems,
references.authors: Michael I Jordan Zoubin Ghahramani Tommi S Jaakkola Lawrence K Saul
references.title: An introduction to variational methods for graphical models
references.venue: Machine learning,
references.authors: Łukasz Kaiser Mohammad Babaeizadeh Piotr Miłos Błażej Osiński Roy H Campbell Konrad Czechowski Dumitru Erhan Chelsea Finn Piotr Kozakowski Sergey Levine
references.title: Model based reinforcement learning for atari
references.venue: In International Conference on Learning Representations,
references.authors: Nal Kalchbrenner Aäron Oord Karen Simonyan Ivo Danihelka Oriol Vinyals Alex Graves Koray Kavukcuoglu
references.title: Video pixel networks
references.venue: In International Conference on Machine Learning,
references.authors: Tero Karras Timo Aila Samuli Laine Jaakko Lehtinen
references.title: Progressive growing of gans for improved quality, stability, and variation
references.venue: arXiv preprint arXiv:1710.10196,
references.authors: Seung Wook Kim Yuhao Zhou Jonah Philion Antonio Torralba Sanja Fidler
references.title: Learning to simulate dynamic environments with gamegan
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Seung Wook Kim Jonah Philion Antonio Torralba Sanja Fidler
references.title: Drivegan: Towards a controllable high-quality neural simulation, 2021
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Diederik P Kingma Max Welling
references.title: Auto-encoding variational bayes
references.venue: International Conference on Learning Representations (ICLR),
references.authors: Pavel Korshunov Sébastien Marcel
references.title: Deepfakes: a new threat to face recognition? assessment and detection
references.venue: arXiv preprint arXiv:1812.08685,
references.authors: Manoj Kumar Mohammad Babaeizadeh Dumitru Erhan Chelsea Finn Sergey Levine Laurent Dinh Durk Kingma
references.title: Videoflow: A flow-based generative model for video
references.authors: Alex X Lee Richard Zhang Frederik Ebert Pieter Abbeel Chelsea Finn Sergey Levine
references.title: Stochastic adversarial video prediction
references.venue: arXiv preprint arXiv:1804.01523,
references.authors: Wonkwang Lee Whie Jung Han Zhang Ting Chen Jing Yu Koh Thomas Huang Hyungsuk Yoon Honglak Lee Seunghoon Hong
references.title: Revisiting hierarchical approach for persistent long-term video prediction
references.venue: arXiv preprint arXiv:2104.06697,
references.authors: Xiaodan Liang Lisa Lee Wei Dai Eric P Xing
references.title: Dual motion gan for future-flow embedded video prediction
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Xue Lin Qi Zou Xixia Xu Yaping Huang Yi Tian
references.title: Motion-aware feature enhancement network for video prediction
references.venue: IEEE Transactions on Circuits and Systems for Video Technology,
references.authors: Ziwei Liu Raymond A Yeh Xiaoou Tang Yiming Liu Aseem Agarwala
references.title: Video frame synthesis using deep voxel flow
references.venue: In Proceedings of the IEEE International Conference on Computer Vision,
references.authors: Chaochao Lu Michael Hirsch Bernhard Scholkopf
references.title: Flexible spatio-temporal networks for video prediction
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Pauline Luc Aidan Clark Sander Dieleman Diego de Las Casas Yotam Doron Albin Cassirer Karen Simonyan
references.title: Transformation-based adversarial video prediction on large-scale data
references.authors: Michael Mathieu Camille Couprie Yann LeCun
references.title: Deep multi-scale video prediction beyond mean square error
references.venue: arXiv preprint arXiv:1511.05440,
references.authors: Matthias Minderer Chen Sun Ruben Villegas Forrester Cole Kevin Murphy Honglak Lee
references.title: Unsupervised learning of object structure and dynamics from videos
references.authors: Guillaume Le Moing Jean Ponce Cordelia Schmid
references.title: Ccvs: Context-aware controllable video synthesis
references.venue: arXiv preprint arXiv:2107.08037,
references.authors: Suraj Nair Chelsea Finn
references.title: Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation
references.authors: Suraj Nair Mohammad Babaeizadeh Chelsea Finn Sergey Levine Vikash Kumar
references.title: Trass: Time reversal as self-supervision
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Seonghyeon Nam Chongyang Ma Menglei Chai William Brendel Ning Xu Seon Joo Kim
references.title: End-to-end time-lapse video synthesis from a single outdoor image
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Junhyuk Oh Xiaoxiao Guo Honglak Lee Richard L Lewis Satinder Singh
references.title: Action-conditional video prediction using deep networks in atari games
references.venue: In Advances in neural information processing systems,
references.authors: Sergiu Oprea Pablo Martinez-Gonzalez Alberto Garcia-Garcia John Alejandro Castro-Vargas Sergio OrtsEscolano Jose Garcia-Rodriguez Antonis Argyros
references.title: A review on deep learning techniques for video prediction
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Chris Paxton Yotam Barnoy Kapil Katyal Raman Arora Gregory D Hager
references.title: Visual robot task planning
references.venue: In 2019 International Conference on Robotics and Automation (ICRA),
references.authors: Rafael Rafailov Tianhe Yu Aravind Rajeswaran Chelsea Finn
references.title: Offline reinforcement learning from images with latent space models
references.venue: arXiv preprint arXiv:2012.11547,
references.authors: Ruslan Rakhimov Denis Volkhonskiy Alexey Artemov Denis Zorin Evgeny Burnaev
references.title: Latent video transformer
references.venue: arXiv preprint arXiv:2006.10704,
references.authors: Prajit Ramachandran Barret Zoph Quoc V Le
references.title: Searching for activation functions
references.venue: arXiv preprint arXiv:1710.05941,
references.authors: MarcAurelio Ranzato Arthur Szlam Joan Bruna Michael Mathieu Ronan Collobert Sumit Chopra
references.title: Video (language) modeling: a baseline for generative models of natural videos
references.venue: arXiv preprint arXiv:1412.6604,
references.authors: Scott E. Reed Aäron van den Oord Nal Kalchbrenner Sergio Gomez Colmenarejo Ziyu Wang Yutian Chen Dan Belov Nando de Freitas
references.title: Parallel multiscale autoregressive density estimation
references.venue: International Conference on Machine Learning (ICML),
references.authors: Danilo Jimenez Rezende Shakir Mohamed Daan Wierstra
references.title: Stochastic backpropagation and approximate inference in deep generative models
references.venue: In International conference on machine learning,
references.authors: Reuven Y Rubinstein
references.title: Optimization of computer simulation models with rare events
references.venue: European Journal of Operational Research,
references.authors: Masaki Saito Shunta Saito
references.title: Tganv2: Efficient training of large models for video generation with multiple subsampling layers
references.authors: Vaibhav Saxena Jimmy Ba Danijar Hafner
references.title: Clockwork variational autoencoders for video prediction
references.venue: arXiv preprint arXiv:2102.09532,
references.authors: Rui Shu James Brofos Frank Zhang Hung Hai Bui Mohammad Ghavamzadeh Mykel Kochenderfer
references.title: Stochastic video prediction with conditional density estimation
references.venue: In ECCV Workshop on Action and Anticipation for Visual Learning,
references.authors: Nitish Srivastava Elman Mansimov Ruslan Salakhudinov
references.title: Unsupervised learning of video representations using lstms
references.venue: In International Conference on Machine Learning,
references.authors: Serkan Sulun A Murat Tekalp
references.title: Can learned frame prediction compete with block motion compensation for video coding
references.venue: Signal, Image and Video Processing,
references.authors: Thomas Unterthiner Sjoerd van Steenkiste Karol Kurach Raphael Marinier Marcin Michalski Sylvain Gelly
references.title: Towards accurate generative models of video: A new metric & challenges
references.venue: arXiv preprint arXiv:1812.01717,
references.authors: Arash Vahdat Jan Kautz
references.title: Nvae: A deep hierarchical variational autoencoder
references.venue: arXiv preprint arXiv:2007.03898,
references.authors: Joost Van Amersfoort Anitha Kannan Marc’Aurelio Ranzato Arthur Szlam Du Tran Soumith Chintala
references.title: Transformation-based models of video sequences
references.venue: arXiv preprint arXiv:1701.08435,
references.authors: Ruben Villegas Jimei Yang Yuliang Zou Sungryull Sohn Xunyu Lin Honglak Lee
references.title: Learning to generate long-term future via hierarchical prediction
references.venue: In international conference on machine learning,
references.authors: Ruben Villegas Arkanath Pathak Harini Kannan Dumitru Erhan Quoc V Le Honglak Lee
references.title: High fidelity video prediction with large stochastic recurrent neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Carl Vondrick Antonio Torralba
references.title: Generating the future with adversarial transformers
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Carl Vondrick Hamed Pirsiavash Antonio Torralba
references.title: Generating videos with scene dynamics
references.venue: arXiv preprint arXiv:1609.02612,
references.authors: Jacob Walker Abhinav Gupta Martial Hebert
references.title: Dense optical flow prediction from a static image
references.venue: In Proceedings of the IEEE International Conference on Computer Vision, pp
references.authors: Jacob Walker Carl Doersch Abhinav Gupta Martial Hebert
references.title: An uncertain future: Forecasting from static images using variational autoencoders
references.venue: In European Conference on Computer Vision,
references.authors: Jacob Walker Ali Razavi Aäron van den Oord
references.title: Predicting video with vqvae
references.venue: arXiv preprint arXiv:2103.01950,
references.authors: Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Guilin Liu Andrew Tao Jan Kautz Bryan Catanzaro
references.title: Video-to-video synthesis
references.venue: arXiv preprint arXiv:1808.06601,
references.authors: Ting-Chun Wang Ming-Yu Liu Andrew Tao Guilin Liu Jan Kautz Bryan Catanzaro
references.title: Few-shot video-tovideo synthesis
references.venue: arXiv preprint arXiv:1910.12713,
references.authors: Yunbo Wang Jianjin Zhang Hongyu Zhu Mingsheng Long Jianmin Wang Philip S Yu
references.title: Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Zhou Wang Alan C Bovik Hamid R Sheikh Eero P Simoncelli
references.title: Image quality assessment: from error visibility to structural similarity
references.venue: IEEE transactions on image processing,
references.authors: Dirk Weissenborn Oscar Täckström Jakob Uszkoreit
references.title: Scaling autoregressive video models
references.venue: arXiv preprint arXiv:1906.02634,
references.authors: Nevan Wichers Ruben Villegas Dumitru Erhan Honglak Lee
references.title: Hierarchical long-term video prediction without supervision
references.venue: International Conference on Machine Learning (ICML),
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault Rémi Louf Morgan Funtowicz
references.title: Huggingface’s transformers: State-of-the-art natural language processing
references.authors: Bohan Wu Suraj Nair Roberto Martin-Martin Li Fei-Fei Chelsea Finn
references.title: Greedy hierarchical variational autoencoders for large-scale video prediction
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Chenfei Wu Lun Huang Qianxi Zhang Binyang Li Lei Ji Fan Yang Guillermo Sapiro Nan Duan
references.title: Godiva: Generating open-domain videos from natural descriptions, 2021b
references.authors: Haixu Wu Zhiyu Yao Mingsheng Long Jianmin Wan
references.title: Motionrnn: A flexible model for video prediction with spacetime-varying motions
references.venue: arXiv preprint arXiv:2103.02243,
references.authors: Annie Xie Frederik Ebert Sergey Levine Chelsea Finn
references.title: Improvisation through physical understanding: Using novel objects as tools with visual foresight
references.venue: In Robotics: Science and Systems (RSS),
references.authors: Tianfan Xue Jiajun Wu Katherine Bouman Bill Freeman
references.title: Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks
references.venue: In Advances in neural information processing systems,
references.authors: Wilson Yan Yunzhi Zhang Pieter Abbeel Aravind Srinivas
references.title: Videogpt: Video generation using vq-vae and transformers
references.venue: arXiv preprint arXiv:2104.10157,
references.authors: Fu-En Yang Jing-Cheng Chang Yuan-Hao Lee Yu-Chiang Frank Wang
references.title: Dual-mtgan: Stochastic and deterministic motion transfer for image-to-video synthesis
references.venue: In 2020 25th International Conference on Pattern Recognition (ICPR),
references.authors: Marvin Zhang Sharad Vikram Laura Smith Pieter Abbeel Matthew Johnson Sergey Levine
references.title: Solar: Deep structured representations for model-based reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Richard Zhang Phillip Isola Alexei A Efros Eli Shechtman Oliver Wang
references.title: The unreasonable effectiveness of deep features as a perceptual metric
references.authors: Denton Fergus
references.title: The encoded output is first averaged across spatial dimension and then decoded into h-size using a fully connected layer. Then, the dynamics are modeled by two LSTM layers. Finally, the output is mapped and reshaped to an image tensor before passing to the decoder. The posterior uses
references.venue: FitVid Dynamics Architecture
references.authors: Ionescu
references.title: License BAIR robot pushing
references.venue: Ebert et al
references.authors: Wu
references.title: Zero-shot robot domain. We evaluate FitVid on planning tasks on a Franka Emika Panda involving kitchen, cleaning, and office items. We did not collect any training data for this task. Instead the model is trained on RoboNet and fine-tuned on augmented data
references.authors: Villegas
references.title: The proposed methods in these papers are also changing the human test subject into a training subject (moslt visible in the changed shirt.)
sections.text: 1 INTRODUCTION
  Predicting what happens next is a cornerstone of intelligence and one of the key capabilities of humans, which we heavily rely on to make decisions in everyday life (Bubic et al., 2010). This capability enables us to anticipate future events and plan ahead to perform temporally extended tasks. While the machine learning literature has studied a wide range of prediction problems, one of the most direct challenges is to predict raw sensory inputs. In particular, prediction of future visual inputs conditioned on a context of past observations – i.e., pixel-level video prediction – encapsulates the challenges of visual perception, modeling of physical events, and reasoning about uncertain behaviors. Video prediction can be formulated as a self-supervised problem, enabling us to use a substantial amount of unlabeled data to provide autonomous systems with powerful predictive capabilities as well as learning rich representations for downstream tasks. Already, video
  models have been successfully deployed in applications such as robotics (Finn & Levine, 2017; Zhang et al., 2019), simulation (Kim et al., 2021; 2020), compression (Duan et al., 2020; Sulun & Tekalp, 2021) and video synthesis from a single frame (Endo et al., 2019; Nam et al., 2019).
  Despite recent advances in generative models in many domains, such as images (Karras et al., 2017; Vahdat & Kautz, 2020) and text (Devlin et al., 2018; Brown et al., 2020), video prediction is still considered to be challenging. Current state-of-the-art methods are limited to low-resolution videos (typically 64×64 (Villegas et al., 2019) and a maximum of 256×256 (Kim et al., 2021)), usually in a narrow domain, such as a single human walking Ionescu et al. (2014), or a robotic arm pushing objects in a stationary setting Ebert et al. (2017). Even then the quality of predicted frames tends to drop substantially after as little as 10 seconds into the future (Saxena et al., 2021).
  A growing body of evidence suggests that underfitting is one of the primary reasons for low quality predictions. For example, Villegas et al. (2019) demonstrate how scaling the model, by adding more parameters, can substantially improve the prediction quality. Similarly, Castrejon et al. (2019) argue that blurry video prediction of variational methods is a sign of underfitting, exhibiting an improved test and train performance as the network capacity increases. Wu et al. (2021a) also observed monotonic improvement as the number of modules in a hierarchical architecture increases. While scaling up models is common to address underfitting, it comes at the cost of more computation, memory, and integration risks (Wolf et al., 2019) as well as more complicated training regimes (Wu et al., 2021a).
  In this paper we take a step back and address underfitting by instead finding an architecture that uses its parameters more efficiently. We propose FitVid, a model that – with the same parameter count as current state-of-the-art models – can fit the current video prediction benchmarks successfully. In fact, FitVid fits these benchmarks so well that in many cases we find it actually begins to suffer from overfitting. To the best of our knowledge, this is the first time a video model reports substantial overfitting on theses benchmarks. Importantly, we also find that simple existing image augmentation techniques can mitigate this overfitting, leading to models that can both fit the training set and generalize well to held-out videos.
  As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics. Furthermore, while each of the components of our proposed model are not novel on their own, we find that FitVid can be trained a significantly simpler training recipe. Prior works on video prediction, particularly those that make use of variational methods to provide for stochasticity, typically require a number of delicate design decisions to train successfully: curriculum training (Oh et al., 2015; Lee et al., 2018; Cenzato et al., 2019), a learned prior (Denton & Fergus, 2018; Castrejon et al., 2019) and annealing of the weight on the VAE KL-divergence penalty (Babaeizadeh et al., 2018). In contrast, we show that our method actually fits the training data well without any such components, training directly via optimizing the evidence lower bound with minimal hyperparameters.
sections.heading: 2 RELATED WORK
sections.text: Video prediction (Ranzato et al., 2014; Srivastava et al., 2015) has been formulated in different ways such as generating videos from a single image (Endo et al., 2019; Nam et al., 2019; Dorkenwald et al., 2021; Yang et al., 2021; Hu et al., 2021) or no image (Vondrick et al., 2016; Saito & Saito, 2018; Clark et al., 2019), text to video generation (Wu et al., 2021b), video-to-video translation (Wang et al., 2019a; 2018) and data-driven simulation (Kaiser et al., 2019; Kim et al., 2021; 2020). In this paper, our focus is on conditional video prediction, which is to predict the future frames of a video conditioned on a few initial context frames and possibly the future actions of the agents (Finn et al., 2016a; Babaeizadeh et al., 2018). Conditional video prediction has a number of applications, including model-based reinforcement learning from pixels (Hafner et al., 2018; 2019; Kaiser et al., 2019; Rafailov et al., 2020) and robotics (Boots et al., 2014; Kalchbrenner et al., 2017a; Ebert et al., 2017; 2018a;b; Xie et al., 2019; Paxton et al., 2019; Nair & Finn, 2020; Nair et al., 2020).
  Initially, video prediction was tackled using deterministic models (Walker et al., 2015; Jia et al., 2016; Xue et al., 2016; Walker et al., 2016; Liang et al., 2017; Byravan & Fox, 2017; Vondrick & Torralba, 2017; Van Amersfoort et al., 2017; Liu et al., 2017; Chen et al., 2017; Lu et al., 2017). Later on, given the common randomness and partial observability in the real-life situations, various stochastic models were proposed to capture the stochasticity of the future. Generative adversarial networks (GANs) (Goodfellow et al., 2014) are used for video predictions (Mathieu et al., 2015; Lee et al., 2018; Clark et al., 2019; Luc et al., 2020; Hong et al., 2021) as sell as autoregressive models (Kalchbrenner et al., 2017b; Reed et al., 2017; Weissenborn et al., 2019). Flow based generative models (Dinh et al., 2014; 2016) are also adopted for video prediction (Kumar et al., 2019).
  In this paper we focus on Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) which are widely used for conditional video prediction (Shu et al., 2016; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Wichers et al., 2018; Villegas et al., 2019; Franceschi et al., 2020; Castrejon et al., 2019; Yan et al., 2021; Rakhimov et al., 2020; Lee et al., 2021; Wu et al., 2021a; Walker et al., 2021). Recently, it has been demonstrated that underfitting on training data plays a major role in low quality blurry predictions in VAE based models. Villegas et al. (2019); Castrejon et al. (2019); Wu et al. (2021a) all reported improved prediction quality as the network capacity increases. In this paper, we are interested in addressing underfitting without increasing the capacity of the current models, and instead, find a more expressive architecture which uses its capacity more efficiently.
sections.heading: 3 THE FITVID MODEL
sections.text: Striving for simplicity, we propose the FitVid model for stochastic video prediction, a convolutional non-hierarchical variational model with a fixed prior. In this section we will describe the proposed model and arcitecture which is also visualized in Fig 2. The appendix includes more architecture details and the code is available in Supplementary Material.
sections.heading: 3.1 VARIATIONAL VIDEO PREDICTION WITH FIXED PRIOR
sections.text: Following prior work (Finn et al., 2016a; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Rubinstein, 1997; Oprea et al., 2020; Wu et al., 2021a), we define the problem of pixel-level video prediction as follows: given the first c frame of a video x<c = x0,x1, . . . ,xc−1, our goal is to predict the future frames by sampling from p(xc:T |x<c). Optionally, the predictive model can be conditioned on additional given information at, such as the actions that the agents in the video are planning to take. This is typically called action-conditioned video prediction.
  Variational video prediction (Babaeizadeh et al., 2018) follows the variational auto-encoder (Kingma & Welling, 2014) formalism by introducing a set of latent variables z to capture the inherent stochasticity of the problem. We factorize the likelihood model to ∏T t=c pθ(xt|x<t, z≤t) which is parametrized in an autoregressive manner over time; i.e. at each timestep t the video frame xt and the latent variables zt are conditioned on the past latent samples and frames.
  For inference, we need to compute a marginalized distribution over the latent variables z, which is intractable. To overcome this problem, we use variation inference (Jordan et al., 1999) by defining an amortized approximate posterior q(z|x) = ∏ t q(zt|z<t,x≤t) that approximates the posterior distribution p(z|x). The approximated posterior is commonly modeled with an inference network qφ(z|x) that outputs the parameters of a conditionally Gaussian distribution N (µφ(x), σφ(x)). This network can be trained using the reparameterization trick (Kingma & Welling, 2014) by optimizing the variational lower bound (Kingma & Welling, 2014; Rezende et al., 2014): L(x) = −Eqφ(z|x) [ log pθ(xt:T |x<t, z) ] + βDKL ( qφ(z|x)||p(z) ) where DKL is the Kullback-Leibler divergence between the approximated posterior and the fixed prior p(z) = N (0, I). The hyper-parameter β represents the trade-off between minimizing frame prediction error and fitting the prior (Higgins et al., 2016b;a; Denton & Fergus, 2018). For FitVid, we set β = 1.
sections.heading: 3.2 THE PROPOSED ARCHITECTURE
sections.text: Encoder and decoder. Following the recent advances in image generation, we use similar residual encoding and decoding cells as NVAE (Vahdat & Kautz, 2020). Each cell includes convolutional
  layers with batch-normalization (Ioffe & Szegedy, 2015) and swish (Ramachandran et al., 2017; Elfwing et al., 2018; Hendrycks & Gimpel, 2016) as the activation function, followed by Squeeze and Excite (Hu et al., 2018). The encoder is made of four encoding blocks with two cells in each block. There is down-sampling after each encoder block using a strided convolution of size three in the spatial dimensions. The decoder also consists of four decoding blocks with two cells in each block, and a nearest neighbour up-sampling after each block. The number of filters in each encoding block is doubled while the number of filters in each decoding block is halved from the previous one. There is a residual skip connection between the encoder and the decoder after each cell which are fixed to the output from the last context frame. The statistics for batch-normalization is averaged across time.
  Dynamics model. Similar to Denton & Fergus (2018), the encoded frame ht is used to predict ht+1 using two layers of LSTMs (Hochreiter & Schmidhuber, 1997). Likewise, q(zt|x<t) is also modeled using a single layer of LSTMs with ht+1 as the input that outputs the parameters of a conditionally Gaussian distribution N (µφ(x), σφ(x)). During the training, z is sampled from q(zt|x<t) while at the inference time z is sampled from the fixed prior N (0, I). The input to the model is always the ground-truth image (which is usually referred to as teacher-forcing (Goodfellow et al., 2016; Cenzato et al., 2019)). At inference time, the predicted image in the previous time-step is used as input to predict the next frame.
  Data augmentation. We find that FitVid can fit even very large datasets very well. However, in many cases, even for datasets that prior models struggle to fit effectively, we find that FitVid actually begins to suffer from overfitting (read Section 4 and 5). To prevent the model from overfitting we use data augmentation. To the best of our knowledge, this is the first use of augmentation in video prediction, perhaps because prior state-of-the-art models tend to underfit already and therefore would not benefit from it. Given the rich literature in image augmentation, we augment the videos using RandAugment (Cubuk et al., 2020). We randomize the augmentation per video but keep the randomization constant for frames of a single video. RandAugment substantially improves the overfitting, however not entirely, as it can be seen in Figure 3. We improve the augmentation by selecting a random crop of the video before resizing it to the desired resolution at the training time, called RandCrop. The combination of RandCrop and RandAugment successfully prevents the overfitting, leading to models that both fit the training set and generalize well to held-out videos.
  What FitVid does not need. Prior works on variational video prediction (Finn et al., 2016b; Babaeizadeh et al., 2018; Lee et al., 2018; Villegas et al., 2019; Wu et al., 2021a), generally require a range of additional design decisions for effective training. Common design parameters include using curriculum training, commonly by scheduled sampling (Bengio et al., 2015), to mitigate distributional shift between training and generation time (Oh et al., 2015; Finn et al., 2016a; Lee et al., 2018; Cenzato et al., 2019); heuristically tuning β in L(x) to balance the prediction vs fitting the prior (Denton & Fergus, 2018) by annealing it over the course of training (Babaeizadeh et al., 2018; Lee et al., 2018) or learned priors (Denton & Fergus, 2018; Villegas et al., 2019; Castrejon et al., 2019; Wu et al., 2021a). Each of these design choices introduces hyperparameters, tuning burden, and additional work when applying a model to a new task. FitVid does not require any of these details: we simply train FitVid by optimizing L(x) using Adam (Kingma & Ba, 2014).
sections.heading: 4 EXPERIMENTS
sections.text: To evaluate FitVid, we test it on four different real-world datasets and compare its performance with prior state-of-the-art methods, with comparable parameter count, using four different metrics. Our main goal is to demonstrate that FitVid can in fact fit on these datasets well and illustrate how augmentation can prevent FitVid from overfitting, resulting in state-of-the-art prediction performance. Please visit the anonymized website sites.google.com/view/fitvid to see samples of generated videos. The code to reproduce these experiments is available in Supplementary Material.
sections.heading: 4.1 EXPERIMENTATION SETUP
sections.text: Datasets: To test FitVid, we use four datasets that cover a variety of real-life scenarios. We use the Human3.6M dataset (Ionescu et al., 2014), which consists of actors performing various actions in a room to study the structured motion prediction. We also use the KITTI dataset (Geiger et al., 2013b) to evaluate FitVid’s ability to handle partial observability and dynamic backgrounds. For
  both datasets, we followed the pre-processing and testing format proposed by Wu et al. (2021a) and Villegas et al. (2019), which predicts 25-frames conditioned the previous five at 64×64 resolution. To evaluate FitVid in an action-conditioned setting, we use RoboNet dataset (Dasari et al., 2019). This large dataset includes more than 15 million video frames from 7 different robotic arms pushing objects in different bins. It contains a wide range of conditions, including different viewpoints, objects, tables, and lighting. Prior video prediction methods have a tendency to badly underfit on this dataset (Dasari et al., 2019). We follow the same setup as Wu et al. (2021a) and randomly select 256 videos for testing. Similar to Wu et al. (2021a), we train FitVid to predict next ten frames given two context frames as well as the ten future actions.
  Finally, to compare FitVid to a wider range of prior work, we use the BAIR robot pushing dataset (Ebert et al., 2017), which is a widely-used benchmark in the video prediction literature. We follow the evaluation protocol of Rakhimov et al. (2020), which predicts the next 16 frames given only one context frame and no actions. Given the high stochasticity of robotic arm movement in BAIR, and the variety of unseen objects in the test set, it is a great benchmark for evaluating the model’s ability to generate diverse outputs.
  Metrics: We evaluate our method and prior models across four different metrics: Structural Similarity Index Measure (SSIM) (Wang et al., 2004), Peak Signal-to-noise Ratio (PSNR) (Huynh-Thu & Ghanbari, 2008), Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) and Fréchet Video Distance (FVD) (Unterthiner et al., 2018). FVD measures the overall visual quality and temporal coherence without reference to the ground truth video. PSNR, SSIM, and LPIPS measure pixel-wise similarity to the ground-truth with LPIPS most accurately representing human perceptual similarity. Given the stochastic nature of video prediction benchmarks, we follow the standard stochastic video prediction evaluation protocol (Babaeizadeh et al., 2018; Villegas et al., 2019; Wu et al., 2021a): we sample 100 future trajectories per video and pick the best one as the final score for PSNR, SSIM and LPIPS. For FVD, we use all 100 with a batch size of 256.
sections.heading: 4.2 RESULTS
sections.text: Comparisons: First, we compare FitVid to GHVAE (Wu et al., 2021a) and SVG (Villegas et al., 2019). GHVAE is a hierarchical variational video prediction model trained in a greedy manner. SVG is a large-scale variational video prediction model with learned prior and minimal inductive bias. We chose these two baseline because they both investigated overfitting by scaling the model, and achieve state-of-the-art results. However, SVG reported no overfitting even for their biggest model ((M = 3,K = 5)) with 298M parameters (Villegas et al., 2019) while GHVAE (with 599M paramteres) reported "some overfitting" on smaller datasets (Wu et al., 2021a). At the same time, both of these models share a similar architecture to FitVid.
  Table 1 contains the results of these experiments. As it can be seen in this table, FitVid outperforms both SVG and GHVAE across all metrics in Robonet and Human3.6M. In KITTI, FitVid also consistently outperforms SVG while either improves or closely matches the performance of GHVAE which has more than twice as parameters. For qualitative results, see Figures 4, 5 and 6.
  Comparison to non-variational methods: To provide a comprehensive comparison with prior methods, including non-variational models, we test FitVid on BAIR dataset (Ebert et al., 2017). As can be seen in Table 2, FitVid outperforms most of the previous models in this setting while performing comparably to Video Transformer (Weissenborn et al., 2019) with 373M parameters.
sections.heading: 5 ANALYSIS
sections.text: In this section, we take a closer look at the results from Section 4, to analyse the consequences of overfitting and the effect of regularization on the current benchmarks.
  On Human3.6M as a video prediction benchmark: Human3.6M (Ionescu et al., 2014) is a common benchmark in video prediction literature (Finn et al., 2016a; Babaeizadeh et al., 2018; Villegas et al., 2017; Wang et al., 2019b; Villegas et al., 2019; Lin et al., 2020; Franceschi et al., 2020; Guen & Thome, 2020; Wu et al., 2021a;c) which we also use to evaluate FitVid (Figure 6). At the first glance, it seems that the model is generating extremely detailed and human-like motions conditioned on the given context pose. However, on closer inspection, we observe that the human subject in the predicted video is changing. In fact, FitVid replaces the unseen human subject into a training subject which is particularly evident from the clothing. Actually, we can find similar video clips from the training data for each one of the predicted videos (see Figure 6). These frames are not exactly the same, but they look notably similar. This observation indicates that:
  1. The model can generalize to unseen frames, poses and subjects since the test context frames are new and unseen. FitVid correctly distinguished the subject from the background, detects the human pose and continues the video from there. 2. The model memorized the motion and the appearance of the training subjects. The model morphs the test human subject into a training one, and then plays a relevant video from the memory.
  This means that FitVid fails to generalize to a new subject, while still generalizing to unseen frames. Given that the Human3.6M has five training and two test subjects (Finn et al., 2016a; Villegas et al., 2017) this may not be surprising.
  Nevertheless, this observation shows how the current low-resolution setup for Human3.6M is not suitable for large-scale video prediction. In fact, after this observation, we traced the same behaviour in other video prediction literature and, unfortunately, it seems this is a common and overlooked issue. For example, the same phenomena can be seen in Figure 6 from Franceschi et al. (2020) that shows changing the test to a training subject by Struct-VRNN (Minderer et al., 2019) and the proposed method by Franceschi et al. (2020) (note the changed shirt). For others examples, see Figure 7 of Villegas et al. (2019) and Figure 5 of Villegas et al. (2017). See Figure 15 for a copy of these figures.
  Overfitting and regularization: As mentioned in Section 1, there is considerable evidence that current video prediction models tend to underfit when trained on large datasets (Villegas et al., 2019; Castrejon et al., 2019; Wu et al., 2021a). Wu et al. (2021a), which is the current state-of-the-art model with 599 million parameters, reported "some overfitting" on smaller datasets such as Human3.6M and KITTI. However, we observe severe and clear overfitting with FitVid, despite having only 302 million parameters. Figure 7 visualizes the training and evaluation LPIPS metric while training FitVid on Human3.6M, without augmentation. This graph demonstrates that the training keeps getting better while the test quality starts to get worse after ∼15K iterations. We also observed similar behaviour on KITTI, as can be seen in Figure 7b. These results clearly shows that FitVid is overfitting on Human3.6M and KITTI, indicating that FitVid is using its parameters more efficiently. As mentioned
  1st Context Frame t = 1
  t = 4 t = 7 t = 10 1st Context Frame t = 1 t = 4 t = 7 t = 10
  1st Context Frame t = 1
  t = 9 t = 17 t = 25 1st Context Frame t = 1 t = 9 t = 17 t = 25
  More videos can be found at sites.google.com/view/fitvid.
  in Section 3.2, to address overfitting, we use augmentation which results in state-of-the-art prediction quality, as reported in Table 1.
  Overfitting on Robonet: We did not observe any overfitting on RoboNet, which is expected given the fact that RoboNet is much larger compared to the other benchmarks. Trying to find a model that can overfit on RoboNet, we test a scaled version of FitVid with 500M parameters – which is still smaller compared to GHVAE with 599M parameters and reported no overfitting on this dataset. This scaled version of FitVid overfits on RoboNet, as demonstrated in Figure 7d. Note that we did not use this scaled version in the reported numbers of Table 1, which is generated using the 302M version. Our goal here was to demonstrate that a scaled version of FitVid can also use its parameters more efficiently, compared to prior models, leading to overfitting on even bigger datasets such as RoboNet.
  Effect of Augmentation on SVG: There is a discrepancy between the input data for training the models in Section 4. FitVid is trained with augmentation while the baselines are trained without any augmentation which raises a question: can the better performance of FitVid be explained only by the augmentation? In other words, do the previous methods benefit from augmentation too? To answer this question, we retrain SVG with and without augmentation. As demonstrated in Table 3, SVG performs worse if trained with augmented data, supporting the claim that it is underfitting to the raw data. As a result, this experiment provides more support for FitVid truly overfitting on these datasets
  and therefore benefiting from augmentation. Please note that we included our SVG results without augmentation too, as we could not perfectly reproduce the numbers reported by Villegas et al. (2019).
  Ablation Study In order to figure out how much each design element in FitVid is contributing to its performamce, we ablate it and check its performance on RoboNet. In this study we try to cover a variety of ablations including encoder/decoder ablations, adding or removing modeling features such as learned prior as well as various skip connection implementations. As can be seen in Table 4, skip connections are a crucial component for archiving high prediction performance. This table also demonstrates that, while modeling sotchasticity is importnat even in action conditioned settings, FitVid does
  not benefit from a learned prior. For more ablation studies, please look at Section A.1.
  Zero-shot Real Robot Performance: Prior work indicate that improved video prediction translates to better performance in the downstream tasks (Wu et al., 2021a; Babaeizadeh et al., 2020). However, in these works, the training and test distribution are the same and there is almost no domain shift from training to testing. In this section, we are interested in investigating whether FitVid is capable of generalizing to a similar but visually different task with no training data for this new domain. Therefore, we setup a real-robot experiment, with a Franka Emika Panda robot arm, in which the goal is to push a specific object to a predetermined goal position. We train FitVid on RoboNet and use cross-entropy method (CEM) (Rubinstein, 1997; Chua et al., 2018) for planning (please see Ap-
  pendix for details). As can be seen in Table 5, this agent is unable to generalize to the new domain, achieving worse performance than a random agent. This may not be surprising given the fact that the videos in RoboNet have entirely different robots and visuals, although the robots are performing the same task (i.e. pushing objects in a bin using a robotic arm).
  We then try to bring the training and test domain closer to each other by fine-tuning FitVid on the data from Wu et al. (2021a). This data contains 5000 autonomously collected videos of a Franka Emika Panda robot arm pushing objects around which look more similar to our setup compared to RoboNet, but still contain different lighting, camera angle, and target objects. This time, we observe that FitVid is relatively successful at generalizing to the new domain, succeeding in 56% of the trials. Finally, we find that adding data augmentation to the fine-tuning improves the generalization ability of the model, achieving 78% success rate. These results illustrate that while large distribution shift adaptation (RoboNet) remains difficult, by using data augmentation FitVid is capable of adapting to a relatively new domain (from Wu et al. (2021a) data).
sections.heading: 6 CONCLUSION
sections.text: We propose FitVid, a simple and scalable variational video prediction model that can attain a significantly better fit to current video prediction datasets even with a similar parameter count as prior models. In fact, while prior methods generally suffer from underfitting on these datasets, naïvely applying FitVid actually results in overfitting. We therefore propose using a set of existing image augmentation techniques that prevent overfitting in video prediction settings, leading to state-of-the-art results across a range of prediction benchmarks.
  To the best of our knowledge, this is the first time a model reports, not only not underfitting, but rather substantial overfitting on these benchmarks. This is particularly important because underfitting is usually cited as one the main reasons for low quality predictions of the future frames. We demonstrate that simple existing image augmentation techniques can be used to prevent the mode from overfitting, resulting in high quality images. In fact, while each of the components of our proposed model are not novel on their own, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics. Moreover, the fit attained by FitVid is so good that a number of previously used benchmarks and metrics can be fooled, resulting in undesired outcomes, which are often overlooked in the video prediction literature.
  There are many ways that FitVid can be expanded. As mentioned in the text, one of the interesting features of our proposed method is that it is simple. It is non-hierarchical, convolutional, with no attention mechanism, no curriculum learning, and no training scheduling. Any of these features can potentially improve the results of FitVid in order to generate even higher quality images. Given the simplicity of FitVid, it can be easily built upon. Another interesting direction would be to introduce new training-aware metrics for video prediction and generation to signal when a model is generating high quality videos by repeating the training data.
sections.heading: ETHICS STATEMENT
sections.text: Videos are an abundant source of visual information about our physical world. They contain information about objects, humans and how they interact with each other. The goal of video prediction is to foremost learn a representation of the world, usable for downstream tasks by an agent. Second, is to predict what happens next, conditioned on the past and the future intents, which can be used for planning. Despite many recent advances in this field, the present day models are still relatively low-quality and limited to narrow domains which makes their applications limited. However, if improved, video prediction or representations learned by video prediction, can be a major step forward toward fully autonomous self-learning agents. This paper, we believe, takes an important step towards this goal by pushing state-of-the-art forward and simplifying it in a meaningful way. However, our model is vulnerable to the bias in the training data and if adopted widely, this can skew research in certain directions. For example, our results may lead to higher quality models which can be scaled to generate even higher quality results. These models will be harder to design and train and require more computational power, and potentially can be biased. Finally, the underlying techniques for video prediction can be misused for generating high-quality videos that are misleading, depicting deliberately false situations or persons, similar to the phenomenon of deepfakes (Korshunov & Marcel, 2018). However, we believe that our experiments are conducted in relatively specific narrow settings and conditions which will likely not generalize broadly. Specifically, it is especially challenging to generate high-quality realistic-looking videos even with such state of the art methods.
sections.heading: REPRODUCIBILITY STATEMENT
sections.text: The anonymized implementation of the model, as well as the code to train and evaluate FitVid is attached in the Supplementary Material. This code can be used to regenerate the results reported in this paper. The approximated computation cost of rerunning these experiments is available in Section A.2.1. Appendix also includes details of implementation such as the training and evaluation details in Section A.2 as well as augmentation details in Algorithm 3. The detailed architecture of the encoder networks can be found in Table 9, decoder in Table 11, and the dynamic model in Table 10 of the Appendix. The hyper-parameters used are reported in Table 8. Table 13 includes the license of the benchmark datasets.
sections.heading: A APPENDIX
sections.heading: A.1 MORE ABLATION STUDIES
sections.text: In order to demonstrate how the design decisions behind FitVid’s architecture affect its performance, we conduct more ablation studies. In the first study, we replace FitVid’s encoder and decoder blocks with the blocks from SVG, while keeping everything else the same, and report the prediction performance on RoboNet in Table 6. From this table, it can be seen that the new encoder and decoder blocks are much more efficient at modeling the videos, resulting in improved prediction metrics. Please note that the numbers reported for SVG are different from Table 1, because we used FitVid ’s dynamic model in this study.
  In another study, we analyze the effect of the number of encoding and decoding blocks on the performance of the model. We very the number of blocks from one to three and measure the prediction quality on RoboNet. As it can be seen in Table 7, it’s not just about making the architecture bigger, but that we are actually benefiting from the particular design of our encoder and decoder i.e. having exactly two encoding blocks and two decoding blocks is the optimal spot for achieving the best results in a reasonable number of training iterations while maintaining the number of parameters in the same ballpark as previous work.
  A.2 IMPLEMENTATION DETAILS
  In this section we describe the details of FitVid’s architecture as well as training. Algorithm 1 and Algorithm 2 describe the high-level training and prediction process respectively. Look at Table 8 for the used hyper-parameters We used the same set of hyper-parameters across all experiments. As mentioned in Section 3.2, all of the hyperparameters are fixed during the training and there is no scheduling. Table 9 and Table 11 include the detailed architecture of the encoder and the decoder. Table 10 describes the structure of dynamic and posterior networks. Finally, Table 12 describes the augmentations details.
sections.heading: A.2.1 COMPUTATION RESOURCES
sections.text: We implement FitVid using Flax (Heek et al., 2020) library for JAX (Bradbury et al., 2018). We train FitVid on 4×4 TPUs (32 co-processors). Each training step (with global batch size of 128 or local batch size of 4) takes ∼ 1020 milliseconds (i.e. 0.98 step per second). In parallel, we evaluate the model every 1000 training steps on a single V100 GPU which takes about ∼ 850 milliseconds for a batch size of 128. The models are trained for one million training iterations which takes ∼ 12 days to complete.
  Algorithm 1: FitVid training. Input: Number of context frames c Data: Training frames x0:T and actions a0:T // Encode all frames
  1 for t← 0 to T do 2 ht,Ct ←, Encoder(xt) 3 end // Prediction 4 sp, sd ← 0,0 // Initialize states 5 for t← 0 to T do
  // Approximate posterior
  6 (µt, σt), sp ← Posterior([ht+1], sp) 7 zt ∼ N (µt, σt)
  // Predict future state
  8 ĥt, sd ← Dynamic([ht, at, zt], sd) 9 end // Decode all frames
  10 for t← 0 to T do // Use last available skip
  connection
  11 x̂t ← Decoder(ĥt, Cc) 12 end
  // Optimize ELBO 13 L ← ||x− x̂||2 +DKL ( N (µ, σ),N (0, I) ) 14 w ←Adam ( w,L )
  Algorithm 2: FitVid prediction. Input: Context frames x0:c Input: All actions a0:T Output: x̂
  1 sd ← 0 // Initialize states 2 ,C←Encoder(xc) // Get last skips. 3 for t← 0 to T do
  // Encode frame
  4 ht,Ct ←, Encoder(xt) // Sample from prior 5 zt ∼ N (N (0, I)) // Predict future state
  6 ĥt, sd ← Dynamic([ht, at, zt], sd) // Decode frame
  7 x̂t ← Decoder(ĥt, Cc) 8 end
  Table 11: FitVid Encoder Architecture. We are using the same encoding and decoding cells as NVAE (Vahdat & Kautz, 2020). The strides are always 1×1. For up-sampling we use nearest neighbour. (bn) is batch-normalization (Ioffe & Szegedy, 2015), (swish) is the activation (Ramachandran et al., 2017), (s&e) is Squeeze and Excite (Hu et al., 2018). There is a skip connection from the beginning of each cell to the end of it. There are also skip connections from each encoder block to the corresponding decoder block (look at Figure 2). In these skip connections, the number of input filters will be matched by the output using a 1×1 convolution.
  Cell Input Size Pre Kernel Filters Post Up Sampling
  1-1 8×8 bn 1×1 2048 - - 1-1 8×8 bn + swish 5×5 2048 - - 1-1 8×8 bn + swish 1×1 512 bn + s&e - 1-2 8×8 bn 1×1 2048 - - 1-2 8×8 bn + swish 5×5 2048 - - 1-2 8×8 bn + swish 1×1 512 bn + s&e Yes 2-1 16×16 bn 1×1 1024 - - 2-1 16×16 bn + swish 5×5 1024 - - 2-1 16×16 bn + swish 1×1 256 bn + s&e - 2-2 16×16 bn 1×1 1024 - - 2-2 16×16 bn + swish 5×5 1024 - - 2-2 16×16 bn + swish 1×1 256 bn + s&e Yes 3-1 32×32 bn 1×1 512 - - 3-1 32×32 bn + swish 5×5 512 - - 3-1 32×32 bn + swish 1×1 128 bn + s&e - 3-2 32×32 bn 1×1 512 - - 3-2 32×32 bn + swish 5×5 512 - - 3-2 32×32 bn + swish 1×1 128 bn + s&e Yes 4-1 64×64 bn 1×1 256 - - 4-1 64×64 bn + swish 5×5 256 - - 4-1 64×64 bn + swish 1×1 64 bn + s&e - 4-2 64×64 bn 1×1 256 - - 4-2 64×64 bn + swish 5×5 256 - - 4-2 64×64 bn + swish 1×1 64 bn + s&e - - 64×64 - 1×1 3 sigmoid -
  Table 12: To prevent FitVid from overfitting we use augmentation. First, at training time, we select a random crop of the video before resizing it to the desired resolution (64×64) at the training time, called RandCrop. This processes crops all the frames of a given video to include a minimum of C percent of the frame’s height. Then we use RandAugment (Cubuk et al., 2020) to improve the augmentation. We use the same augmentation configuration for all the datasets. Per video, we use the same randomization across all the frames.
  Algorithm 3: Video Augmentation. Input: Video x Input: Number of RandAugment transformations N Input: RandAugment magnitude M Input: RandCrop crop height minimum ratio C
  Hyper-parameter Value
  RandAugment
  No. of transformationsN 1 MagnitudeM 5
  Transformations
  Dataset Reference License
  RoboNet Dasari et al. (2019) MIT KITTI Geiger et al. (2013a) Creative Commons Human3.6M Ionescu et al. (2014) License BAIR robot pushing Ebert et al. (2017) GitHub Default
sections.heading: A.3 ROBOT EXPERIMENT DETAILS
sections.heading: A.3.1 ENVIRONMENT
sections.text: The robot environment consists of a Franka Emika Panda robot operating over a bin which contains various objects. The robot’s observations are 64×64×3 RGB images, and its action space consists of 3 DOF delta position control of the end effector with action magnitudes in the range [-10cm,10cm].
  A.3.2 DATA
  The data used for finetuning the model used in robot experiments was taken directly from Wu et al. (2021a). This data consists of different viewpoint, lighting conditions, and target objects than what is used in our evaluation.
sections.heading: A.3.3 EVALUATION DETAILS
sections.text: During evaluation, the agent is specified to complete an object pushing task by a goal image. The agent has 50 timesteps to complete the task, and a trial is measured as successful if the majority of the object overlaps with its target position at some point in the episode. Each method is evaluated over 18 trials, of which 6 consist of objects in an “office” setting, 6 consist of objects in a “kitchen” setting, and 6 consist of objects in a “cleaning” setting (See Figure 9).
  To execute the task, the agent performs visual model predictive control using the cross entropy method (CEM). Specifically, the agent takes in 1 frame, and predicts trajectories of 10 time-steps for 200 different sampled action sequences. Trajectories are ranked according to their negative mean squared error to the goal image, averaged across all 10 time-steps. The action distribution refits to the top 20 actions, and repeats for 3 iterations of CEM. Afterwards the best sequence of 10 actions is stepped in the environment in an open loop faction. The process repeats 5 times until the end of the episode.
  Ground Truth1s t C
  on te
  xt F
  ra m
  e t=
  1
  Predicted
  t= 2
  t= 3
  t= 4
  t= 5
  t= 6
  t= 7
  t= 8
  t= 9
  t= 10
  Ground Truth
  Predicted
  Fi gu
  re 11
  :M or
  e de
  ta ile
  d vi
  de o
  fr om
  Fi gu
  re 4
  w hi
  ch ill
  us tra
  te s
  Fi tV
  id on
  ac tio
  nco
  nd iti
  on ed
  R ob
  oN et
  (D as
  ar ie
  ta l.,
  20 19
  ). Th
  e m
  od el
  is co
  nd iti
  on ed
  on th
  e fir
  st tw o fr am es an d is pr ed ic tin g th e ne xt te n fr am es gi ve n th e fu tu re ac tio ns of th e ro bo tic ar m .T he se fig ur es de m on st ra te ho w th e pr ed ic te d m ov em en ts of th e ar m cl os el y fo llo w s th e gr ou nd tru th gi ve n th at th e fu tu re ac tio ns is kn ow n. Th e m od el al so pr ed ic ts de ta ile d m ov em en ts of th e pu sh ed ob je ct s (v is ib le in th e to p ex am pl e) as w el l as fil lin g in th e pr ev io us ly un se en ba ck gr ou nd w ith so m e ra nd om ob je ct s (l oo k at th e ob je ct th at ap pe ar be hi nd th e ro bo tic ar m in th e bo tto m ex am pl e) .A ls o no tic e th e w ro ng pr ed ic tio ns of ro bo ts fin ge rs in th e bo tto m ex am pl e.
  Ground Truth1s t C
  on te
  xt F
  ra m
  e t=
  1
  Predicted
  t= 3
  t= 5
  t= 7
  t= 9
  t= 11
  t= 13
  t= 15
  t= 17
  t= 19
  t= 21
  t= 23
  t= 25
  Ground Truth
  Predicted
  Fi gu
  re 12
  :M or
  e de
  ta ile
  d vi
  de o
  fr om
  Fi gu
  re 5
  w hi
  ch ill
  us tr
  at es
  Fi tV
  id on
  K IT
  T Id
  at as
  et (G
  ei ge
  re ta
  l., 20
  13 b)
  .A s
  it ca
  n be
  se en
  in th
  is fig
  ur e,
  th e
  m od
  el ge
  ne ra
  te s
  hi gh
  qu al
  ity pr
  ed ic
  tio n
  of th
  e fu
  tu re
  in a
  dy na
  m ic
  sc en
  e. N
  ot e
  ho w
  in th
  e to
  p ex
  am pl
  e Fi
  tV id
  ke ep
  s pr
  ed ic
  tin g
  th e
  m ov
  em en
  to ft
  he sh
  ad ow
  on th
  e gr
  ou nd
  til li
  tm ov es ou to ft he fr am e. A ft er th at ,t he m od el ke ep s pu sh in g th e ba ck gr ou nd cl os er in ea ch fr am e, im pl yi ng dr iv in g fo rw ar d. W e no tic ed th at th e qu al ity of pr ed ic tio ns dr op su bs ta nt ia lly fa st er w he n th er e ar e m or e ob je ct s in th e sc en e e. g. th e dr iv in g sc en es in si de a ci ty as ca n be se en in th e bo tto m ex am pl e. T hi s in di ca te s th e m od el st ill fa ils to ge ne ra liz e to m or e co m pl ex sc en es w ith m or e m ov in g su bj ec ts .
  Ground Truth1s t C
  on te
  xt F
  ra m
  e t=
  1
  Predicted
  t= 3
  t= 5
  t= 7
  t= 9
  t= 11
  t= 13
  t= 15
  t= 17
  t= 19
  t= 21
  t= 23
  t= 25
  Ground Truth
  Predicted
  Fi gu
  re 13
  : M
  or e
  de ta
  ile d
  vi de
  o fr
  om Fi
  gu re
  6 w
  hi ch
  ill us
  tr at
  es Fi
  tV id
  on H
  um an
  3. 6M
  (I on
  es cu
  et al
  ., 20
  14 ).
  T hi
  s fig
  ur e
  de m
  on st
  ra te
  s ex
  tr em
  el y
  de ta
  ile d
  an d
  hu m
  an -l
  ik e
  m ot
  io ns
  pr ed
  ic te
  d by
  Fi tV
  id ,c
  on di
  tio ne
  d on
  th e
  gi ve
  n co
  nt ex
  tf ra
  m es
  .H ow
  ev er
  ,o n
  cl os
  er in
  sp ec
  tio n,
  it ca
  n be
  se en
  th at
  th e
  hu m
  an su
  bj ec
  ti n
  th e
  vi de o is ch an gi ng ,f ro m th e te st su bj ec tt o a tr ai ni ng su bj ec t. T hi s is pa rt ic ul ar ly ev id en tf ro m th e cl ot hs .T hi s ph en om en a in di ca te s th at ,a lth ou gh Fi tV id is ca pa bl e of ge ne ra liz in g to th e fr am es ou to ft ra in in g di st ri bu tio n, ho w ev er ,i tm or ph s th e hu m an su bj ec ti nt o a fa m ili ar on e fr om th e tr ai ni ng se ta nd th en pl ay s th e vi de o fr om th e m em or y.
  Ground TruthC on
  te xt
  F ra
  m e
  t= 1 Predicted t=
  3 t=
  5 t=
  7 t=
  9 t=
  11 t=
  13 t=
  15
  Ground Truth
  Predicted
  Fi gu
  re 14
  :M or
  e de
  ta ile
  d vi
  de o
  fr om
  Fi gu
  re 8
  w hi
  ch ill
  us tr
  at es
  Fi tV
  id on
  B A
  IR ro
  bo tp
  us hi
  ng da
  ta se
  t( E
  be rt
  et al
  ., 20
  17 )w
  ith no
  ac tio
  ns .T
  he m
  od el
  is co
  nd iti
  on ed
  on ly
  on th
  e fir
  st fr
  am e
  an d
  is pr
  ed ic
  tin g
  th e
  ne xt
  16 fr
  am es
  .G iv
  en th
  at th
  e fu
  tu re
  ac tio
  ns of
  th e
  ro bo
  tic ar
  m is
  un kn
  ow n,
  th e
  pr ed
  ic tio
  n ca
  n di
  ve rg
  e su
  bs ta
  nt ia lly fr om th e gr ou nd tr ut h vi de o. H ow ev er ,t he m od el pr ed ic ts m ov em en ts fo rt he ob je ct s w he ne ve rt he ar m pu sh es th e ob je ct in an im ag in ar y sc en ar io .I ta ls o fil ls th e ba ck gr ou nd w ith ra nd om ob je ct s.
title: FITVID: HIGH-CAPACITY PIXEL-LEVEL VIDEO PREDICTION