summaries: The paper studies conditional video prediction. In particular, it focuses on the problem of current models underfitting and not scaling to datasets. The paper proposes an architecture that is claimed capable of using parameters more efficiently in order to overfit. Then, data augmentation is introduced to improve performance for generalization. They evaluated their method against baselines on 4 datasets (Human 3.6M, KITTI, Robonet, BAIR pushing dataset). They also showed experiments to support overfitting/underfitting claims.

summaries: The paper proposes a simple and scalable variational video prediction model FitVid, which attains a better fit to video prediction datasets even with a similar parameter count as prior models. The author has observed that previous methods suffer from underfitting on these datasets, directly applying FitVid actually results in overfitting. The FitVid uses a set of existing image augmentation techniques to prevent overfitting so that it can achieve state-of-the-art results on several prediction benchmarks. FitVid's architecture is based on SE-UNet LSTM, which seems to be the common backbone for the stochastic video prediction task.

summaries: This paper presents a new network architecture called FitVid to perform the task of video prediction, i.e. the task of predicting future frames from previous frames. Previous methods for this tasks usually suffer from "underfitting", while FitVid is able to overfit on major benchmarks without increasing the number of parameters, thanks to a better model architecture. What is more, FitVid is much easier to train than previous work, without using any bells and whistles in training. As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics.

summaries: This paper discusses a new framework named FITVID to handle the problem of video prediction. FitVid is built on existing modules like Sikp connection, Sequeeze and Excite, and LSTMs. FitVid only needs a simple training strategy and begins overfitting the datasets. Data augmentation is also adopted to handle the problem of overfitting.

abstractText: An agent that is capable of predicting what happens next can perform a variety of tasks through planning with no additional training. Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for many other visual perception tasks. This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances. Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain. There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions. In this paper, we argue that the inefficient use of parameters in the current video models is the main reason for underfitting. Therefore, we introduce a new architecture, named FitVid, which is capable of fitting the common benchmarks so well that it begins to suffer from overfitting – while having similar parameter count as the current state-of-theart models. We analyze the consequences of overfitting, illustrating how it can produce unexpected outcomes such as generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques. As a result, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics.

authors:
Mohammad Babaeizadeh
Chelsea Finn
Dumitru Erhan
Roy H Campbell
Sergey Levine

title: Stochastic variational video prediction

authors:
Mohammad Babaeizadeh
Mohammad Taghi Saffar
Danijar Hafner
Harini Kannan
Chelsea Finn
Sergey Levine
Dumitru Erhan

title: Models, pixels, and rewards: Evaluating design trade-offs in visual model-based reinforcement learning

authors:
Samy Bengio
Oriol Vinyals
Navdeep Jaitly
Noam Shazeer

title: Scheduled sampling for sequence prediction with recurrent neural networks

venue: arXiv preprint arXiv:1506.03099,

authors:
Byron Boots
Arunkumar Byravan
Dieter Fox

title: Learning predictive models of a depth camera & manipulator from raw execution traces

venue: IEEE International Conference on Robotics and Automation (ICRA),

authors:
James Bradbury
Roy Frostig
Peter Hawkins
Matthew James Johnson
Chris Leary
Dougal Maclaurin
George Necula
Adam Paszke
Jake VanderPlas
Skye Wanderman-Milne
Qiao Zhang

title: JAX: composable transformations of Python+NumPy programs, 2018

venue: URL http://github.com/google/jax

authors:
Andreja Bubic
D Yves Von Cramon
Ricarda I Schubotz

title: Prediction, cognition and the brain

venue: Frontiers in human neuroscience,

authors:
Arunkumar Byravan
Dieter Fox

title: Se3-nets: Learning rigid body motion using deep neural networks

authors:
Lluis Castrejon
Nicolas Ballas
Aaron Courville

title: Improved conditional vrnns for video prediction

venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,

authors:
Alberto Cenzato
Alberto Testolin
Marco Zorzi

title: On the difficulty of learning and predicting the long-term dynamics of bouncing objects

authors:
Baoyang Chen
Wenmin Wang
Jinzhuo Wang

title: Video imagination from a single image with transformation generation

venue: In Proceedings of the on Thematic Workshops of ACM Multimedia 2017,

authors:
Kurtland Chua
Roberto Calandra
Rowan McAllister
Sergey Levine

title: Deep reinforcement learning in a handful of trials using probabilistic dynamics models

venue: In Advances in Neural Information Processing Systems,

authors:
Aidan Clark
Jeff Donahue
Karen Simonyan

title: Adversarial video generation on complex datasets

venue: arXiv preprint arXiv:1907.06571,

authors:
Ekin D Cubuk
Barret Zoph
Jonathon Shlens
Quoc V Le

title: Randaugment: Practical automated data augmentation with a reduced search space

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,

authors:
Sudeep Dasari
Frederik Ebert
Stephen Tian
Suraj Nair
Bernadette Bucher
Karl Schmeckpeper
Siddharth Singh
Sergey Levine
Chelsea Finn

title: Robonet: Large-scale multi-robot learning

venue: CoRL,

authors:
Emily Denton
Rob Fergus

title: Stochastic video generation with a learned prior

venue: arXiv preprint arXiv:1802.07687,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
Laurent Dinh
David Krueger
Yoshua Bengio

title: Nice: Non-linear independent components estimation

venue: arXiv preprint arXiv:1410.8516,

authors:
Laurent Dinh
Jascha Sohl-Dickstein
Samy Bengio

title: Density estimation using real nvp

venue: arXiv preprint arXiv:1605.08803,

authors:
Michael Dorkenwald
Timo Milbich
Andreas Blattmann
Robin Rombach
Konstantinos G Derpanis
Bjorn Ommer

title: Stochastic image-to-video synthesis using cinns

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Lingyu Duan
Jiaying Liu
Wenhan Yang
Tiejun Huang
Wen Gao

title: Video coding for machines: A paradigm of collaborative compression and intelligent analytics

venue: IEEE Transactions on Image Processing,

authors:
Frederik Ebert
Chelsea Finn
Alex X. Lee
Sergey Levine

title: Self-Supervised Visual Planning with Temporal Skip Connections

venue: Conference on Robot Learning (CoRL),

authors:
Frederik Ebert
Chelsea Finn
Alex X Lee
Sergey Levine

title: Self-supervised visual planning with temporal skip connections

venue: In Conference on Robot Learning (CoRL),

authors:
Frederik Ebert
Sudeep Dasari
Alex X Lee
Sergey Levine
Chelsea Finn

title: Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning

authors:
Frederik Ebert
Chelsea Finn
Sudeep Dasari
Annie Xie
Alex Lee
Sergey Levine

title: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control

venue: arXiv preprint arXiv:1812.00568,

authors:
Stefan Elfwing
Eiji Uchibe
Kenji Doya

title: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

venue: Neural Networks,

authors:
Yuki Endo
Yoshihiro Kanamori
Shigeru Kuriyama

title: Animating landscape: self-supervised learning of decoupled motion and appearance for single-image video synthesis

authors:
Chelsea Finn
Sergey Levine

title: Deep visual foresight for planning robot motion

authors:
Chelsea Finn
Ian Goodfellow
Sergey Levine

title: Unsupervised learning for physical interaction through video prediction

venue: In Advances in neural information processing systems,

authors:
Chelsea Finn
Xin Yu Tan
Yan Duan
Trevor Darrell
Sergey Levine
Pieter Abbeel

title: Deep spatial autoencoders for visuomotor learning

authors:
Jean-Yves Franceschi
Edouard Delasalles
Mickaël Chen
Sylvain Lamprier
Patrick Gallinari

title: Stochastic latent residual video prediction

venue: arXiv preprint arXiv:2002.09219,

authors:
Andreas Geiger
Philip Lenz
Christoph Stiller
Raquel Urtasun

title: Vision meets robotics: The kitti dataset

venue: International Journal of Robotics Research (IJRR),

venue: The International Journal of Robotics Research,

authors:
Ian J Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial networks

venue: arXiv preprint arXiv:1406.2661,

authors:
Vincent Le Guen
Nicolas Thome

title: Disentangling physical dynamics from unknown factors for unsupervised video prediction

authors:
Danijar Hafner
Timothy Lillicrap
Ian Fischer
Ruben Villegas
David Ha
Honglak Lee
James Davidson

title: Learning latent dynamics for planning from pixels

venue: arXiv preprint arXiv:1811.04551,

authors:
Danijar Hafner
Timothy Lillicrap
Jimmy Ba
Mohammad Norouzi

title: Dream to control: Learning behaviors by latent imagination

venue: arXiv preprint arXiv:1912.01603,

authors:
Jonathan Heek
Anselm Levskaya
Avital Oliver
Marvin Ritter
Bertrand Rondepierre
Andreas Steiner
Marc van Zee

title: Flax: A neural network library and ecosystem for JAX

authors:
Dan Hendrycks
Kevin Gimpel

title: Gaussian error linear units (gelus)

venue: arXiv preprint arXiv:1606.08415,

authors:
Irina Higgins
Loic Matthey
Xavier Glorot
Arka Pal
Benigno Uria
Charles Blundell
Shakir Mohamed
Alexander Lerchner

title: Early visual concept learning with unsupervised deep learning

venue: arXiv preprint arXiv:1606.05579,

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: beta-vae: Learning basic visual concepts with a constrained variational framework

venue: International Conference on Learning Representations (ICLR),

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Kibeom Hong
Youngjung Uh
Hyeran Byun

title: Arrowgan: Learning to generate videos by learning arrow of time

venue: arXiv preprint arXiv:2101.03710,

authors:
Jie Hu
Li Shen
Gang Sun

title: Squeeze-and-excitation networks

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Qiyang Hu
Adrian Wälchli
Tiziano Portenier
Matthias Zwicker
Paolo Favaro

title: Learning to take directions one step at a time

venue: In 2020 25th International Conference on Pattern Recognition (ICPR),

authors:
Quan Huynh-Thu
Mohammed Ghanbari

title: Scope of validity of psnr in image/video quality assessment

venue: Electronics letters,

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: In International conference on machine learning,

authors:
Catalin Ionescu
Dragos Papava
Vlad Olaru
Cristian Sminchisescu

title: Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Xu Jia
Bert De Brabandere
Tinne Tuytelaars
Luc V Gool

title: Dynamic filter networks

authors:
Michael I Jordan
Zoubin Ghahramani
Tommi S Jaakkola
Lawrence K Saul

title: An introduction to variational methods for graphical models

venue: Machine learning,

authors:
Łukasz Kaiser
Mohammad Babaeizadeh
Piotr Miłos
Błażej Osiński
Roy H Campbell
Konrad Czechowski
Dumitru Erhan
Chelsea Finn
Piotr Kozakowski
Sergey Levine

title: Model based reinforcement learning for atari

venue: In International Conference on Learning Representations,

authors:
Nal Kalchbrenner
Aäron Oord
Karen Simonyan
Ivo Danihelka
Oriol Vinyals
Alex Graves
Koray Kavukcuoglu

title: Video pixel networks

venue: In International Conference on Machine Learning,

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

venue: arXiv preprint arXiv:1710.10196,

authors:
Seung Wook Kim
Yuhao Zhou
Jonah Philion
Antonio Torralba
Sanja Fidler

title: Learning to simulate dynamic environments with gamegan

authors:
Seung Wook Kim
Jonah Philion
Antonio Torralba
Sanja Fidler

title: Drivegan: Towards a controllable high-quality neural simulation, 2021

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

authors:
Pavel Korshunov
Sébastien Marcel

title: Deepfakes: a new threat to face recognition? assessment and detection

venue: arXiv preprint arXiv:1812.08685,

authors:
Manoj Kumar
Mohammad Babaeizadeh
Dumitru Erhan
Chelsea Finn
Sergey Levine
Laurent Dinh
Durk Kingma

title: Videoflow: A flow-based generative model for video

authors:
Alex X Lee
Richard Zhang
Frederik Ebert
Pieter Abbeel
Chelsea Finn
Sergey Levine

title: Stochastic adversarial video prediction

venue: arXiv preprint arXiv:1804.01523,

authors:
Wonkwang Lee
Whie Jung
Han Zhang
Ting Chen
Jing Yu Koh
Thomas Huang
Hyungsuk Yoon
Honglak Lee
Seunghoon Hong

title: Revisiting hierarchical approach for persistent long-term video prediction

venue: arXiv preprint arXiv:2104.06697,

authors:
Xiaodan Liang
Lisa Lee
Wei Dai
Eric P Xing

title: Dual motion gan for future-flow embedded video prediction

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Xue Lin
Qi Zou
Xixia Xu
Yaping Huang
Yi Tian

title: Motion-aware feature enhancement network for video prediction

venue: IEEE Transactions on Circuits and Systems for Video Technology,

authors:
Ziwei Liu
Raymond A Yeh
Xiaoou Tang
Yiming Liu
Aseem Agarwala

title: Video frame synthesis using deep voxel flow

authors:
Chaochao Lu
Michael Hirsch
Bernhard Scholkopf

title: Flexible spatio-temporal networks for video prediction

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Pauline Luc
Aidan Clark
Sander Dieleman
Diego de Las Casas
Yotam Doron
Albin Cassirer
Karen Simonyan

title: Transformation-based adversarial video prediction on large-scale data

authors:
Michael Mathieu
Camille Couprie
Yann LeCun

title: Deep multi-scale video prediction beyond mean square error

venue: arXiv preprint arXiv:1511.05440,

authors:
Matthias Minderer
Chen Sun
Ruben Villegas
Forrester Cole
Kevin Murphy
Honglak Lee

title: Unsupervised learning of object structure and dynamics from videos

authors:
Guillaume Le Moing
Jean Ponce
Cordelia Schmid

title: Ccvs: Context-aware controllable video synthesis

venue: arXiv preprint arXiv:2107.08037,

authors:
Suraj Nair
Chelsea Finn

title: Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation

authors:
Suraj Nair
Mohammad Babaeizadeh
Chelsea Finn
Sergey Levine
Vikash Kumar

title: Trass: Time reversal as self-supervision

authors:
Seonghyeon Nam
Chongyang Ma
Menglei Chai
William Brendel
Ning Xu
Seon Joo Kim

title: End-to-end time-lapse video synthesis from a single outdoor image

authors:
Junhyuk Oh
Xiaoxiao Guo
Honglak Lee
Richard L Lewis
Satinder Singh

title: Action-conditional video prediction using deep networks in atari games

authors:
Sergiu Oprea
Pablo Martinez-Gonzalez
Alberto Garcia-Garcia
John Alejandro Castro-Vargas
Sergio OrtsEscolano
Jose Garcia-Rodriguez
Antonis Argyros

title: A review on deep learning techniques for video prediction

venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,

authors:
Chris Paxton
Yotam Barnoy
Kapil Katyal
Raman Arora
Gregory D Hager

title: Visual robot task planning

venue: In 2019 International Conference on Robotics and Automation (ICRA),

authors:
Rafael Rafailov
Tianhe Yu
Aravind Rajeswaran
Chelsea Finn

title: Offline reinforcement learning from images with latent space models

venue: arXiv preprint arXiv:2012.11547,

authors:
Ruslan Rakhimov
Denis Volkhonskiy
Alexey Artemov
Denis Zorin
Evgeny Burnaev

title: Latent video transformer

venue: arXiv preprint arXiv:2006.10704,

authors:
Prajit Ramachandran
Barret Zoph
Quoc V Le

title: Searching for activation functions

venue: arXiv preprint arXiv:1710.05941,

authors:
MarcAurelio Ranzato
Arthur Szlam
Joan Bruna
Michael Mathieu
Ronan Collobert
Sumit Chopra

title: Video (language) modeling: a baseline for generative models of natural videos

venue: arXiv preprint arXiv:1412.6604,

authors:
Scott E. Reed
Aäron van den Oord
Nal Kalchbrenner
Sergio Gomez Colmenarejo
Ziyu Wang
Yutian Chen
Dan Belov
Nando de Freitas

title: Parallel multiscale autoregressive density estimation

venue: International Conference on Machine Learning (ICML),

authors:
Danilo Jimenez Rezende
Shakir Mohamed
Daan Wierstra

title: Stochastic backpropagation and approximate inference in deep generative models

title: Optimization of computer simulation models with rare events

venue: European Journal of Operational Research,

authors:
Masaki Saito
Shunta Saito

title: Tganv2: Efficient training of large models for video generation with multiple subsampling layers

authors:
Vaibhav Saxena
Jimmy Ba
Danijar Hafner

title: Clockwork variational autoencoders for video prediction

venue: arXiv preprint arXiv:2102.09532,

authors:
Rui Shu
James Brofos
Frank Zhang
Hung Hai Bui
Mohammad Ghavamzadeh
Mykel Kochenderfer

title: Stochastic video prediction with conditional density estimation

venue: In ECCV Workshop on Action and Anticipation for Visual Learning,

authors:
Nitish Srivastava
Elman Mansimov
Ruslan Salakhudinov

title: Unsupervised learning of video representations using lstms

authors:
Serkan Sulun
A Murat Tekalp

title: Can learned frame prediction compete with block motion compensation for video coding

venue: Signal, Image and Video Processing,

authors:
Thomas Unterthiner
Sjoerd van Steenkiste
Karol Kurach
Raphael Marinier
Marcin Michalski
Sylvain Gelly

title: Towards accurate generative models of video: A new metric & challenges

venue: arXiv preprint arXiv:1812.01717,

authors:
Arash Vahdat
Jan Kautz

title: Nvae: A deep hierarchical variational autoencoder

venue: arXiv preprint arXiv:2007.03898,

authors:
Joost Van Amersfoort
Anitha Kannan
Marc’Aurelio Ranzato
Arthur Szlam
Du Tran
Soumith Chintala

title: Transformation-based models of video sequences

venue: arXiv preprint arXiv:1701.08435,

authors:
Ruben Villegas
Jimei Yang
Yuliang Zou
Sungryull Sohn
Xunyu Lin
Honglak Lee

title: Learning to generate long-term future via hierarchical prediction

venue: In international conference on machine learning,

authors:
Ruben Villegas
Arkanath Pathak
Harini Kannan
Dumitru Erhan
Quoc V Le
Honglak Lee

title: High fidelity video prediction with large stochastic recurrent neural networks

authors:
Carl Vondrick
Antonio Torralba

title: Generating the future with adversarial transformers

authors:
Carl Vondrick
Hamed Pirsiavash
Antonio Torralba

title: Generating videos with scene dynamics

venue: arXiv preprint arXiv:1609.02612,

authors:
Jacob Walker
Abhinav Gupta
Martial Hebert

title: Dense optical flow prediction from a static image

venue: In Proceedings of the IEEE International Conference on Computer Vision, pp

authors:
Jacob Walker
Carl Doersch
Abhinav Gupta
Martial Hebert

title: An uncertain future: Forecasting from static images using variational autoencoders

venue: In European Conference on Computer Vision,

authors:
Jacob Walker
Ali Razavi
Aäron van den Oord

title: Predicting video with vqvae

venue: arXiv preprint arXiv:2103.01950,

authors:
Ting-Chun Wang
Ming-Yu Liu
Jun-Yan Zhu
Guilin Liu
Andrew Tao
Jan Kautz
Bryan Catanzaro

title: Video-to-video synthesis

venue: arXiv preprint arXiv:1808.06601,

authors:
Ting-Chun Wang
Ming-Yu Liu
Andrew Tao
Guilin Liu
Jan Kautz
Bryan Catanzaro

title: Few-shot video-tovideo synthesis

venue: arXiv preprint arXiv:1910.12713,

authors:
Yunbo Wang
Jianjin Zhang
Hongyu Zhu
Mingsheng Long
Jianmin Wang
Philip S Yu

title: Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics

authors:
Zhou Wang
Alan C Bovik
Hamid R Sheikh
Eero P Simoncelli

title: Image quality assessment: from error visibility to structural similarity

venue: IEEE transactions on image processing,

authors:
Dirk Weissenborn
Oscar Täckström
Jakob Uszkoreit

title: Scaling autoregressive video models

venue: arXiv preprint arXiv:1906.02634,

authors:
Nevan Wichers
Ruben Villegas
Dumitru Erhan
Honglak Lee

title: Hierarchical long-term video prediction without supervision

authors:
Thomas Wolf
Lysandre Debut
Victor Sanh
Julien Chaumond
Clement Delangue
Anthony Moi
Pierric Cistac
Tim Rault
Rémi Louf
Morgan Funtowicz

title: Huggingface’s transformers: State-of-the-art natural language processing

authors:
Bohan Wu
Suraj Nair
Roberto Martin-Martin
Li Fei-Fei
Chelsea Finn

title: Greedy hierarchical variational autoencoders for large-scale video prediction

authors:
Chenfei Wu
Lun Huang
Qianxi Zhang
Binyang Li
Lei Ji
Fan Yang
Guillermo Sapiro
Nan Duan

title: Godiva: Generating open-domain videos from natural descriptions, 2021b

authors:
Haixu Wu
Zhiyu Yao
Mingsheng Long
Jianmin Wan

title: Motionrnn: A flexible model for video prediction with spacetime-varying motions

venue: arXiv preprint arXiv:2103.02243,

authors:
Annie Xie
Frederik Ebert
Sergey Levine
Chelsea Finn

title: Improvisation through physical understanding: Using novel objects as tools with visual foresight

venue: In Robotics: Science and Systems (RSS),

authors:
Tianfan Xue
Jiajun Wu
Katherine Bouman
Bill Freeman

title: Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks

authors:
Wilson Yan
Yunzhi Zhang
Pieter Abbeel
Aravind Srinivas

title: Videogpt: Video generation using vq-vae and transformers

venue: arXiv preprint arXiv:2104.10157,

authors:
Fu-En Yang
Jing-Cheng Chang
Yuan-Hao Lee
Yu-Chiang Frank Wang

title: Dual-mtgan: Stochastic and deterministic motion transfer for image-to-video synthesis

authors:
Marvin Zhang
Sharad Vikram
Laura Smith
Pieter Abbeel
Matthew Johnson
Sergey Levine

title: Solar: Deep structured representations for model-based reinforcement learning

authors:
Richard Zhang
Phillip Isola
Alexei A Efros
Eli Shechtman
Oliver Wang

title: The unreasonable effectiveness of deep features as a perceptual metric

authors: Denton Fergus

title: The encoded output is first averaged across spatial dimension and then decoded into h-size using a fully connected layer. Then, the dynamics are modeled by two LSTM layers. Finally, the output is mapped and reshaped to an image tensor before passing to the decoder. The posterior uses

venue: FitVid Dynamics Architecture

title: License BAIR robot pushing

venue: Ebert et al

title: Zero-shot robot domain. We evaluate FitVid on planning tasks on a Franka Emika Panda involving kitchen, cleaning, and office items. We did not collect any training data for this task. Instead the model is trained on RoboNet and fine-tuned on augmented data

title: The proposed methods in these papers are also changing the human test subject into a training subject (moslt visible in the changed shirt.)

text:
1 INTRODUCTION
Predicting what happens next is a cornerstone of intelligence and one of the key capabilities of humans, which we heavily rely on to make decisions in everyday life (Bubic et al., 2010). This capability enables us to anticipate future events and plan ahead to perform temporally extended tasks. While the machine learning literature has studied a wide range of prediction problems, one of the most direct challenges is to predict raw sensory inputs. In particular, prediction of future visual inputs conditioned on a context of past observations – i.e., pixel-level video prediction – encapsulates the challenges of visual perception, modeling of physical events, and reasoning about uncertain behaviors. Video prediction can be formulated as a self-supervised problem, enabling us to use a substantial amount of unlabeled data to provide autonomous systems with powerful predictive capabilities as well as learning rich representations for downstream tasks. Already, video
models have been successfully deployed in applications such as robotics (Finn & Levine, 2017; Zhang et al., 2019), simulation (Kim et al., 2021; 2020), compression (Duan et al., 2020; Sulun & Tekalp, 2021) and video synthesis from a single frame (Endo et al., 2019; Nam et al., 2019).
Despite recent advances in generative models in many domains, such as images (Karras et al., 2017; Vahdat & Kautz, 2020) and text (Devlin et al., 2018; Brown et al., 2020), video prediction is still considered to be challenging. Current state-of-the-art methods are limited to low-resolution videos (typically 64×64 (Villegas et al., 2019) and a maximum of 256×256 (Kim et al., 2021)), usually in a narrow domain, such as a single human walking Ionescu et al. (2014), or a robotic arm pushing objects in a stationary setting Ebert et al. (2017). Even then the quality of predicted frames tends to drop substantially after as little as 10 seconds into the future (Saxena et al., 2021).
A growing body of evidence suggests that underfitting is one of the primary reasons for low quality predictions. For example, Villegas et al. (2019) demonstrate how scaling the model, by adding more parameters, can substantially improve the prediction quality. Similarly, Castrejon et al. (2019) argue that blurry video prediction of variational methods is a sign of underfitting, exhibiting an improved test and train performance as the network capacity increases. Wu et al. (2021a) also observed monotonic improvement as the number of modules in a hierarchical architecture increases. While scaling up models is common to address underfitting, it comes at the cost of more computation, memory, and integration risks (Wolf et al., 2019) as well as more complicated training regimes (Wu et al., 2021a).
In this paper we take a step back and address underfitting by instead finding an architecture that uses its parameters more efficiently. We propose FitVid, a model that – with the same parameter count as current state-of-the-art models – can fit the current video prediction benchmarks successfully. In fact, FitVid fits these benchmarks so well that in many cases we find it actually begins to suffer from overfitting. To the best of our knowledge, this is the first time a video model reports substantial overfitting on theses benchmarks. Importantly, we also find that simple existing image augmentation techniques can mitigate this overfitting, leading to models that can both fit the training set and generalize well to held-out videos.
As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics. Furthermore, while each of the components of our proposed model are not novel on their own, we find that FitVid can be trained a significantly simpler training recipe. Prior works on video prediction, particularly those that make use of variational methods to provide for stochasticity, typically require a number of delicate design decisions to train successfully: curriculum training (Oh et al., 2015; Lee et al., 2018; Cenzato et al., 2019), a learned prior (Denton & Fergus, 2018; Castrejon et al., 2019) and annealing of the weight on the VAE KL-divergence penalty (Babaeizadeh et al., 2018). In contrast, we show that our method actually fits the training data well without any such components, training directly via optimizing the evidence lower bound with minimal hyperparameters.

heading: 2 RELATED WORK

text:
Video prediction (Ranzato et al., 2014; Srivastava et al., 2015) has been formulated in different ways such as generating videos from a single image (Endo et al., 2019; Nam et al., 2019; Dorkenwald et al., 2021; Yang et al., 2021; Hu et al., 2021) or no image (Vondrick et al., 2016; Saito & Saito, 2018; Clark et al., 2019), text to video generation (Wu et al., 2021b), video-to-video translation (Wang et al., 2019a; 2018) and data-driven simulation (Kaiser et al., 2019; Kim et al., 2021; 2020). In this paper, our focus is on conditional video prediction, which is to predict the future frames of a video conditioned on a few initial context frames and possibly the future actions of the agents (Finn et al., 2016a; Babaeizadeh et al., 2018). Conditional video prediction has a number of applications, including model-based reinforcement learning from pixels (Hafner et al., 2018; 2019; Kaiser et al., 2019; Rafailov et al., 2020) and robotics (Boots et al., 2014; Kalchbrenner et al., 2017a; Ebert et al., 2017; 2018a;b; Xie et al., 2019; Paxton et al., 2019; Nair & Finn, 2020; Nair et al., 2020).
Initially, video prediction was tackled using deterministic models (Walker et al., 2015; Jia et al., 2016; Xue et al., 2016; Walker et al., 2016; Liang et al., 2017; Byravan & Fox, 2017; Vondrick & Torralba, 2017; Van Amersfoort et al., 2017; Liu et al., 2017; Chen et al., 2017; Lu et al., 2017). Later on, given the common randomness and partial observability in the real-life situations, various stochastic models were proposed to capture the stochasticity of the future. Generative adversarial networks (GANs) (Goodfellow et al., 2014) are used for video predictions (Mathieu et al., 2015; Lee et al., 2018; Clark et al., 2019; Luc et al., 2020; Hong et al., 2021) as sell as autoregressive models (Kalchbrenner et al., 2017b; Reed et al., 2017; Weissenborn et al., 2019). Flow based generative models (Dinh et al., 2014; 2016) are also adopted for video prediction (Kumar et al., 2019).
In this paper we focus on Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) which are widely used for conditional video prediction (Shu et al., 2016; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Wichers et al., 2018; Villegas et al., 2019; Franceschi et al., 2020; Castrejon et al., 2019; Yan et al., 2021; Rakhimov et al., 2020; Lee et al., 2021; Wu et al., 2021a; Walker et al., 2021). Recently, it has been demonstrated that underfitting on training data plays a major role in low quality blurry predictions in VAE based models. Villegas et al. (2019); Castrejon et al. (2019); Wu et al. (2021a) all reported improved prediction quality as the network capacity increases. In this paper, we are interested in addressing underfitting without increasing the capacity of the current models, and instead, find a more expressive architecture which uses its capacity more efficiently.

heading: 3 THE FITVID MODEL

text: Striving for simplicity, we propose the FitVid model for stochastic video prediction, a convolutional non-hierarchical variational model with a fixed prior. In this section we will describe the proposed model and arcitecture which is also visualized in Fig 2. The appendix includes more architecture details and the code is available in Supplementary Material.

heading: 3.1 VARIATIONAL VIDEO PREDICTION WITH FIXED PRIOR

text:
Following prior work (Finn et al., 2016a; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Rubinstein, 1997; Oprea et al., 2020; Wu et al., 2021a), we define the problem of pixel-level video prediction as follows: given the first c frame of a video x<c = x0,x1, . . . ,xc−1, our goal is to predict the future frames by sampling from p(xc:T |x<c). Optionally, the predictive model can be conditioned on additional given information at, such as the actions that the agents in the video are planning to take. This is typically called action-conditioned video prediction.
Variational video prediction (Babaeizadeh et al., 2018) follows the variational auto-encoder (Kingma & Welling, 2014) formalism by introducing a set of latent variables z to capture the inherent stochasticity of the problem. We factorize the likelihood model to ∏T t=c pθ(xt|x<t, z≤t) which is parametrized in an autoregressive manner over time; i.e. at each timestep t the video frame xt and the latent variables zt are conditioned on the past latent samples and frames.
For inference, we need to compute a marginalized distribution over the latent variables z, which is intractable. To overcome this problem, we use variation inference (Jordan et al., 1999) by defining an amortized approximate posterior q(z|x) = ∏ t q(zt|z<t,x≤t) that approximates the posterior distribution p(z|x). The approximated posterior is commonly modeled with an inference network qφ(z|x) that outputs the parameters of a conditionally Gaussian distribution N (µφ(x), σφ(x)). This network can be trained using the reparameterization trick (Kingma & Welling, 2014) by optimizing the variational lower bound (Kingma & Welling, 2014; Rezende et al., 2014): L(x) = −Eqφ(z|x) [ log pθ(xt:T |x<t, z) ] + βDKL ( qφ(z|x)||p(z) ) where DKL is the Kullback-Leibler divergence between the approximated posterior and the fixed prior p(z) = N (0, I). The hyper-parameter β represents the trade-off between minimizing frame prediction error and fitting the prior (Higgins et al., 2016b;a; Denton & Fergus, 2018). For FitVid, we set β = 1.

heading: 3.2 THE PROPOSED ARCHITECTURE

text:
Encoder and decoder. Following the recent advances in image generation, we use similar residual encoding and decoding cells as NVAE (Vahdat & Kautz, 2020). Each cell includes convolutional
layers with batch-normalization (Ioffe & Szegedy, 2015) and swish (Ramachandran et al., 2017; Elfwing et al., 2018; Hendrycks & Gimpel, 2016) as the activation function, followed by Squeeze and Excite (Hu et al., 2018). The encoder is made of four encoding blocks with two cells in each block. There is down-sampling after each encoder block using a strided convolution of size three in the spatial dimensions. The decoder also consists of four decoding blocks with two cells in each block, and a nearest neighbour up-sampling after each block. The number of filters in each encoding block is doubled while the number of filters in each decoding block is halved from the previous one. There is a residual skip connection between the encoder and the decoder after each cell which are fixed to the output from the last context frame. The statistics for batch-normalization is averaged across time.
Dynamics model. Similar to Denton & Fergus (2018), the encoded frame ht is used to predict ht+1 using two layers of LSTMs (Hochreiter & Schmidhuber, 1997). Likewise, q(zt|x<t) is also modeled using a single layer of LSTMs with ht+1 as the input that outputs the parameters of a conditionally Gaussian distribution N (µφ(x), σφ(x)). During the training, z is sampled from q(zt|x<t) while at the inference time z is sampled from the fixed prior N (0, I). The input to the model is always the ground-truth image (which is usually referred to as teacher-forcing (Goodfellow et al., 2016; Cenzato et al., 2019)). At inference time, the predicted image in the previous time-step is used as input to predict the next frame.
Data augmentation. We find that FitVid can fit even very large datasets very well. However, in many cases, even for datasets that prior models struggle to fit effectively, we find that FitVid actually begins to suffer from overfitting (read Section 4 and 5). To prevent the model from overfitting we use data augmentation. To the best of our knowledge, this is the first use of augmentation in video prediction, perhaps because prior state-of-the-art models tend to underfit already and therefore would not benefit from it. Given the rich literature in image augmentation, we augment the videos using RandAugment (Cubuk et al., 2020). We randomize the augmentation per video but keep the randomization constant for frames of a single video. RandAugment substantially improves the overfitting, however not entirely, as it can be seen in Figure 3. We improve the augmentation by selecting a random crop of the video before resizing it to the desired resolution at the training time, called RandCrop. The combination of RandCrop and RandAugment successfully prevents the overfitting, leading to models that both fit the training set and generalize well to held-out videos.
What FitVid does not need. Prior works on variational video prediction (Finn et al., 2016b; Babaeizadeh et al., 2018; Lee et al., 2018; Villegas et al., 2019; Wu et al., 2021a), generally require a range of additional design decisions for effective training. Common design parameters include using curriculum training, commonly by scheduled sampling (Bengio et al., 2015), to mitigate distributional shift between training and generation time (Oh et al., 2015; Finn et al., 2016a; Lee et al., 2018; Cenzato et al., 2019); heuristically tuning β in L(x) to balance the prediction vs fitting the prior (Denton & Fergus, 2018) by annealing it over the course of training (Babaeizadeh et al., 2018; Lee et al., 2018) or learned priors (Denton & Fergus, 2018; Villegas et al., 2019; Castrejon et al., 2019; Wu et al., 2021a). Each of these design choices introduces hyperparameters, tuning burden, and additional work when applying a model to a new task. FitVid does not require any of these details: we simply train FitVid by optimizing L(x) using Adam (Kingma & Ba, 2014).

heading: 4 EXPERIMENTS

text: To evaluate FitVid, we test it on four different real-world datasets and compare its performance with prior state-of-the-art methods, with comparable parameter count, using four different metrics. Our main goal is to demonstrate that FitVid can in fact fit on these datasets well and illustrate how augmentation can prevent FitVid from overfitting, resulting in state-of-the-art prediction performance. Please visit the anonymized website sites.google.com/view/fitvid to see samples of generated videos. The code to reproduce these experiments is available in Supplementary Material.

heading: 4.1 EXPERIMENTATION SETUP

text:
Datasets: To test FitVid, we use four datasets that cover a variety of real-life scenarios. We use the Human3.6M dataset (Ionescu et al., 2014), which consists of actors performing various actions in a room to study the structured motion prediction. We also use the KITTI dataset (Geiger et al., 2013b) to evaluate FitVid’s ability to handle partial observability and dynamic backgrounds. For
both datasets, we followed the pre-processing and testing format proposed by Wu et al. (2021a) and Villegas et al. (2019), which predicts 25-frames conditioned the previous five at 64×64 resolution. To evaluate FitVid in an action-conditioned setting, we use RoboNet dataset (Dasari et al., 2019). This large dataset includes more than 15 million video frames from 7 different robotic arms pushing objects in different bins. It contains a wide range of conditions, including different viewpoints, objects, tables, and lighting. Prior video prediction methods have a tendency to badly underfit on this dataset (Dasari et al., 2019). We follow the same setup as Wu et al. (2021a) and randomly select 256 videos for testing. Similar to Wu et al. (2021a), we train FitVid to predict next ten frames given two context frames as well as the ten future actions.
Finally, to compare FitVid to a wider range of prior work, we use the BAIR robot pushing dataset (Ebert et al., 2017), which is a widely-used benchmark in the video prediction literature. We follow the evaluation protocol of Rakhimov et al. (2020), which predicts the next 16 frames given only one context frame and no actions. Given the high stochasticity of robotic arm movement in BAIR, and the variety of unseen objects in the test set, it is a great benchmark for evaluating the model’s ability to generate diverse outputs.
Metrics: We evaluate our method and prior models across four different metrics: Structural Similarity Index Measure (SSIM) (Wang et al., 2004), Peak Signal-to-noise Ratio (PSNR) (Huynh-Thu & Ghanbari, 2008), Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) and Fréchet Video Distance (FVD) (Unterthiner et al., 2018). FVD measures the overall visual quality and temporal coherence without reference to the ground truth video. PSNR, SSIM, and LPIPS measure pixel-wise similarity to the ground-truth with LPIPS most accurately representing human perceptual similarity. Given the stochastic nature of video prediction benchmarks, we follow the standard stochastic video prediction evaluation protocol (Babaeizadeh et al., 2018; Villegas et al., 2019; Wu et al., 2021a): we sample 100 future trajectories per video and pick the best one as the final score for PSNR, SSIM and LPIPS. For FVD, we use all 100 with a batch size of 256.

heading: 4.2 RESULTS

text:
Comparisons: First, we compare FitVid to GHVAE (Wu et al., 2021a) and SVG (Villegas et al., 2019). GHVAE is a hierarchical variational video prediction model trained in a greedy manner. SVG is a large-scale variational video prediction model with learned prior and minimal inductive bias. We chose these two baseline because they both investigated overfitting by scaling the model, and achieve state-of-the-art results. However, SVG reported no overfitting even for their biggest model ((M = 3,K = 5)) with 298M parameters (Villegas et al., 2019) while GHVAE (with 599M paramteres) reported "some overfitting" on smaller datasets (Wu et al., 2021a). At the same time, both of these models share a similar architecture to FitVid.
Table 1 contains the results of these experiments. As it can be seen in this table, FitVid outperforms both SVG and GHVAE across all metrics in Robonet and Human3.6M. In KITTI, FitVid also consistently outperforms SVG while either improves or closely matches the performance of GHVAE which has more than twice as parameters. For qualitative results, see Figures 4, 5 and 6.
Comparison to non-variational methods: To provide a comprehensive comparison with prior methods, including non-variational models, we test FitVid on BAIR dataset (Ebert et al., 2017). As can be seen in Table 2, FitVid outperforms most of the previous models in this setting while performing comparably to Video Transformer (Weissenborn et al., 2019) with 373M parameters.

heading: 5 ANALYSIS

text:
In this section, we take a closer look at the results from Section 4, to analyse the consequences of overfitting and the effect of regularization on the current benchmarks.
On Human3.6M as a video prediction benchmark: Human3.6M (Ionescu et al., 2014) is a common benchmark in video prediction literature (Finn et al., 2016a; Babaeizadeh et al., 2018; Villegas et al., 2017; Wang et al., 2019b; Villegas et al., 2019; Lin et al., 2020; Franceschi et al., 2020; Guen & Thome, 2020; Wu et al., 2021a;c) which we also use to evaluate FitVid (Figure 6). At the first glance, it seems that the model is generating extremely detailed and human-like motions conditioned on the given context pose. However, on closer inspection, we observe that the human subject in the predicted video is changing. In fact, FitVid replaces the unseen human subject into a training subject which is particularly evident from the clothing. Actually, we can find similar video clips from the training data for each one of the predicted videos (see Figure 6). These frames are not exactly the same, but they look notably similar. This observation indicates that:
1. The model can generalize to unseen frames, poses and subjects since the test context frames are new and unseen. FitVid correctly distinguished the subject from the background, detects the human pose and continues the video from there. 2. The model memorized the motion and the appearance of the training subjects. The model morphs the test human subject into a training one, and then plays a relevant video from the memory.
This means that FitVid fails to generalize to a new subject, while still generalizing to unseen frames. Given that the Human3.6M has five training and two test subjects (Finn et al., 2016a; Villegas et al., 2017) this may not be surprising.
Nevertheless, this observation shows how the current low-resolution setup for Human3.6M is not suitable for large-scale video prediction. In fact, after this observation, we traced the same behaviour in other video prediction literature and, unfortunately, it seems this is a common and overlooked issue. For example, the same phenomena can be seen in Figure 6 from Franceschi et al. (2020) that shows changing the test to a training subject by Struct-VRNN (Minderer et al., 2019) and the proposed method by Franceschi et al. (2020) (note the changed shirt). For others examples, see Figure 7 of Villegas et al. (2019) and Figure 5 of Villegas et al. (2017). See Figure 15 for a copy of these figures.
Overfitting and regularization: As mentioned in Section 1, there is considerable evidence that current video prediction models tend to underfit when trained on large datasets (Villegas et al., 2019; Castrejon et al., 2019; Wu et al., 2021a). Wu et al. (2021a), which is the current state-of-the-art model with 599 million parameters, reported "some overfitting" on smaller datasets such as Human3.6M and KITTI. However, we observe severe and clear overfitting with FitVid, despite having only 302 million parameters. Figure 7 visualizes the training and evaluation LPIPS metric while training FitVid on Human3.6M, without augmentation. This graph demonstrates that the training keeps getting better while the test quality starts to get worse after ∼15K iterations. We also observed similar behaviour on KITTI, as can be seen in Figure 7b. These results clearly shows that FitVid is overfitting on Human3.6M and KITTI, indicating that FitVid is using its parameters more efficiently. As mentioned
1st Context Frame t = 1
t = 4 t = 7 t = 10 1st Context Frame t = 1 t = 4 t = 7 t = 10
1st Context Frame t = 1
t = 9 t = 17 t = 25 1st Context Frame t = 1 t = 9 t = 17 t = 25
More videos can be found at sites.google.com/view/fitvid.
in Section 3.2, to address overfitting, we use augmentation which results in state-of-the-art prediction quality, as reported in Table 1.
Overfitting on Robonet: We did not observe any overfitting on RoboNet, which is expected given the fact that RoboNet is much larger compared to the other benchmarks. Trying to find a model that can overfit on RoboNet, we test a scaled version of FitVid with 500M parameters – which is still smaller compared to GHVAE with 599M parameters and reported no overfitting on this dataset. This scaled version of FitVid overfits on RoboNet, as demonstrated in Figure 7d. Note that we did not use this scaled version in the reported numbers of Table 1, which is generated using the 302M version. Our goal here was to demonstrate that a scaled version of FitVid can also use its parameters more efficiently, compared to prior models, leading to overfitting on even bigger datasets such as RoboNet.
Effect of Augmentation on SVG: There is a discrepancy between the input data for training the models in Section 4. FitVid is trained with augmentation while the baselines are trained without any augmentation which raises a question: can the better performance of FitVid be explained only by the augmentation? In other words, do the previous methods benefit from augmentation too? To answer this question, we retrain SVG with and without augmentation. As demonstrated in Table 3, SVG performs worse if trained with augmented data, supporting the claim that it is underfitting to the raw data. As a result, this experiment provides more support for FitVid truly overfitting on these datasets
and therefore benefiting from augmentation. Please note that we included our SVG results without augmentation too, as we could not perfectly reproduce the numbers reported by Villegas et al. (2019).
Ablation Study In order to figure out how much each design element in FitVid is contributing to its performamce, we ablate it and check its performance on RoboNet. In this study we try to cover a variety of ablations including encoder/decoder ablations, adding or removing modeling features such as learned prior as well as various skip connection implementations. As can be seen in Table 4, skip connections are a crucial component for archiving high prediction performance. This table also demonstrates that, while modeling sotchasticity is importnat even in action conditioned settings, FitVid does
not benefit from a learned prior. For more ablation studies, please look at Section A.1.
Zero-shot Real Robot Performance: Prior work indicate that improved video prediction translates to better performance in the downstream tasks (Wu et al., 2021a; Babaeizadeh et al., 2020). However, in these works, the training and test distribution are the same and there is almost no domain shift from training to testing. In this section, we are interested in investigating whether FitVid is capable of generalizing to a similar but visually different task with no training data for this new domain. Therefore, we setup a real-robot experiment, with a Franka Emika Panda robot arm, in which the goal is to push a specific object to a predetermined goal position. We train FitVid on RoboNet and use cross-entropy method (CEM) (Rubinstein, 1997; Chua et al., 2018) for planning (please see Ap-
pendix for details). As can be seen in Table 5, this agent is unable to generalize to the new domain, achieving worse performance than a random agent. This may not be surprising given the fact that the videos in RoboNet have entirely different robots and visuals, although the robots are performing the same task (i.e. pushing objects in a bin using a robotic arm).
We then try to bring the training and test domain closer to each other by fine-tuning FitVid on the data from Wu et al. (2021a). This data contains 5000 autonomously collected videos of a Franka Emika Panda robot arm pushing objects around which look more similar to our setup compared to RoboNet, but still contain different lighting, camera angle, and target objects. This time, we observe that FitVid is relatively successful at generalizing to the new domain, succeeding in 56% of the trials. Finally, we find that adding data augmentation to the fine-tuning improves the generalization ability of the model, achieving 78% success rate. These results illustrate that while large distribution shift adaptation (RoboNet) remains difficult, by using data augmentation FitVid is capable of adapting to a relatively new domain (from Wu et al. (2021a) data).

heading: 6 CONCLUSION

text:
We propose FitVid, a simple and scalable variational video prediction model that can attain a significantly better fit to current video prediction datasets even with a similar parameter count as prior models. In fact, while prior methods generally suffer from underfitting on these datasets, naïvely applying FitVid actually results in overfitting. We therefore propose using a set of existing image augmentation techniques that prevent overfitting in video prediction settings, leading to state-of-the-art results across a range of prediction benchmarks.
To the best of our knowledge, this is the first time a model reports, not only not underfitting, but rather substantial overfitting on these benchmarks. This is particularly important because underfitting is usually cited as one the main reasons for low quality predictions of the future frames. We demonstrate that simple existing image augmentation techniques can be used to prevent the mode from overfitting, resulting in high quality images. In fact, while each of the components of our proposed model are not novel on their own, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics. Moreover, the fit attained by FitVid is so good that a number of previously used benchmarks and metrics can be fooled, resulting in undesired outcomes, which are often overlooked in the video prediction literature.
There are many ways that FitVid can be expanded. As mentioned in the text, one of the interesting features of our proposed method is that it is simple. It is non-hierarchical, convolutional, with no attention mechanism, no curriculum learning, and no training scheduling. Any of these features can potentially improve the results of FitVid in order to generate even higher quality images. Given the simplicity of FitVid, it can be easily built upon. Another interesting direction would be to introduce new training-aware metrics for video prediction and generation to signal when a model is generating high quality videos by repeating the training data.

heading: ETHICS STATEMENT

text: Videos are an abundant source of visual information about our physical world. They contain information about objects, humans and how they interact with each other. The goal of video prediction is to foremost learn a representation of the world, usable for downstream tasks by an agent. Second, is to predict what happens next, conditioned on the past and the future intents, which can be used for planning. Despite many recent advances in this field, the present day models are still relatively low-quality and limited to narrow domains which makes their applications limited. However, if improved, video prediction or representations learned by video prediction, can be a major step forward toward fully autonomous self-learning agents. This paper, we believe, takes an important step towards this goal by pushing state-of-the-art forward and simplifying it in a meaningful way. However, our model is vulnerable to the bias in the training data and if adopted widely, this can skew research in certain directions. For example, our results may lead to higher quality models which can be scaled to generate even higher quality results. These models will be harder to design and train and require more computational power, and potentially can be biased. Finally, the underlying techniques for video prediction can be misused for generating high-quality videos that are misleading, depicting deliberately false situations or persons, similar to the phenomenon of deepfakes (Korshunov & Marcel, 2018). However, we believe that our experiments are conducted in relatively specific narrow settings and conditions which will likely not generalize broadly. Specifically, it is especially challenging to generate high-quality realistic-looking videos even with such state of the art methods.

heading: REPRODUCIBILITY STATEMENT

text: The anonymized implementation of the model, as well as the code to train and evaluate FitVid is attached in the Supplementary Material. This code can be used to regenerate the results reported in this paper. The approximated computation cost of rerunning these experiments is available in Section A.2.1. Appendix also includes details of implementation such as the training and evaluation details in Section A.2 as well as augmentation details in Algorithm 3. The detailed architecture of the encoder networks can be found in Table 9, decoder in Table 11, and the dynamic model in Table 10 of the Appendix. The hyper-parameters used are reported in Table 8. Table 13 includes the license of the benchmark datasets.

heading: A APPENDIX

heading: A.1 MORE ABLATION STUDIES

text:
In order to demonstrate how the design decisions behind FitVid’s architecture affect its performance, we conduct more ablation studies. In the first study, we replace FitVid’s encoder and decoder blocks with the blocks from SVG, while keeping everything else the same, and report the prediction performance on RoboNet in Table 6. From this table, it can be seen that the new encoder and decoder blocks are much more efficient at modeling the videos, resulting in improved prediction metrics. Please note that the numbers reported for SVG are different from Table 1, because we used FitVid ’s dynamic model in this study.
In another study, we analyze the effect of the number of encoding and decoding blocks on the performance of the model. We very the number of blocks from one to three and measure the prediction quality on RoboNet. As it can be seen in Table 7, it’s not just about making the architecture bigger, but that we are actually benefiting from the particular design of our encoder and decoder i.e. having exactly two encoding blocks and two decoding blocks is the optimal spot for achieving the best results in a reasonable number of training iterations while maintaining the number of parameters in the same ballpark as previous work.
A.2 IMPLEMENTATION DETAILS
In this section we describe the details of FitVid’s architecture as well as training. Algorithm 1 and Algorithm 2 describe the high-level training and prediction process respectively. Look at Table 8 for the used hyper-parameters We used the same set of hyper-parameters across all experiments. As mentioned in Section 3.2, all of the hyperparameters are fixed during the training and there is no scheduling. Table 9 and Table 11 include the detailed architecture of the encoder and the decoder. Table 10 describes the structure of dynamic and posterior networks. Finally, Table 12 describes the augmentations details.

heading: A.2.1 COMPUTATION RESOURCES

text:
We implement FitVid using Flax (Heek et al., 2020) library for JAX (Bradbury et al., 2018). We train FitVid on 4×4 TPUs (32 co-processors). Each training step (with global batch size of 128 or local batch size of 4) takes ∼ 1020 milliseconds (i.e. 0.98 step per second). In parallel, we evaluate the model every 1000 training steps on a single V100 GPU which takes about ∼ 850 milliseconds for a batch size of 128. The models are trained for one million training iterations which takes ∼ 12 days to complete.
Algorithm 1: FitVid training. Input: Number of context frames c Data: Training frames x0:T and actions a0:T // Encode all frames
1 for t← 0 to T do 2 ht,Ct ←, Encoder(xt) 3 end // Prediction 4 sp, sd ← 0,0 // Initialize states 5 for t← 0 to T do
// Approximate posterior
6 (µt, σt), sp ← Posterior([ht+1], sp) 7 zt ∼ N (µt, σt)
// Predict future state
8 ĥt, sd ← Dynamic([ht, at, zt], sd) 9 end // Decode all frames
10 for t← 0 to T do // Use last available skip
connection
11 x̂t ← Decoder(ĥt, Cc) 12 end
// Optimize ELBO 13 L ← ||x− x̂||2 +DKL ( N (µ, σ),N (0, I) ) 14 w ←Adam ( w,L )
Algorithm 2: FitVid prediction. Input: Context frames x0:c Input: All actions a0:T Output: x̂
1 sd ← 0 // Initialize states 2 ,C←Encoder(xc) // Get last skips. 3 for t← 0 to T do
// Encode frame
4 ht,Ct ←, Encoder(xt) // Sample from prior 5 zt ∼ N (N (0, I)) // Predict future state
6 ĥt, sd ← Dynamic([ht, at, zt], sd) // Decode frame
7 x̂t ← Decoder(ĥt, Cc) 8 end
Table 11: FitVid Encoder Architecture. We are using the same encoding and decoding cells as NVAE (Vahdat & Kautz, 2020). The strides are always 1×1. For up-sampling we use nearest neighbour. (bn) is batch-normalization (Ioffe & Szegedy, 2015), (swish) is the activation (Ramachandran et al., 2017), (s&e) is Squeeze and Excite (Hu et al., 2018). There is a skip connection from the beginning of each cell to the end of it. There are also skip connections from each encoder block to the corresponding decoder block (look at Figure 2). In these skip connections, the number of input filters will be matched by the output using a 1×1 convolution.
Cell Input Size Pre Kernel Filters Post Up Sampling
1-1 8×8 bn 1×1 2048 - - 1-1 8×8 bn + swish 5×5 2048 - - 1-1 8×8 bn + swish 1×1 512 bn + s&e - 1-2 8×8 bn 1×1 2048 - - 1-2 8×8 bn + swish 5×5 2048 - - 1-2 8×8 bn + swish 1×1 512 bn + s&e Yes 2-1 16×16 bn 1×1 1024 - - 2-1 16×16 bn + swish 5×5 1024 - - 2-1 16×16 bn + swish 1×1 256 bn + s&e - 2-2 16×16 bn 1×1 1024 - - 2-2 16×16 bn + swish 5×5 1024 - - 2-2 16×16 bn + swish 1×1 256 bn + s&e Yes 3-1 32×32 bn 1×1 512 - - 3-1 32×32 bn + swish 5×5 512 - - 3-1 32×32 bn + swish 1×1 128 bn + s&e - 3-2 32×32 bn 1×1 512 - - 3-2 32×32 bn + swish 5×5 512 - - 3-2 32×32 bn + swish 1×1 128 bn + s&e Yes 4-1 64×64 bn 1×1 256 - - 4-1 64×64 bn + swish 5×5 256 - - 4-1 64×64 bn + swish 1×1 64 bn + s&e - 4-2 64×64 bn 1×1 256 - - 4-2 64×64 bn + swish 5×5 256 - - 4-2 64×64 bn + swish 1×1 64 bn + s&e - - 64×64 - 1×1 3 sigmoid -
Table 12: To prevent FitVid from overfitting we use augmentation. First, at training time, we select a random crop of the video before resizing it to the desired resolution (64×64) at the training time, called RandCrop. This processes crops all the frames of a given video to include a minimum of C percent of the frame’s height. Then we use RandAugment (Cubuk et al., 2020) to improve the augmentation. We use the same augmentation configuration for all the datasets. Per video, we use the same randomization across all the frames.
Algorithm 3: Video Augmentation. Input: Video x Input: Number of RandAugment transformations N Input: RandAugment magnitude M Input: RandCrop crop height minimum ratio C
Hyper-parameter Value
RandAugment
No. of transformationsN 1 MagnitudeM 5
Transformations
Dataset Reference License
RoboNet Dasari et al. (2019) MIT KITTI Geiger et al. (2013a) Creative Commons Human3.6M Ionescu et al. (2014) License BAIR robot pushing Ebert et al. (2017) GitHub Default

heading: A.3 ROBOT EXPERIMENT DETAILS

heading: A.3.1 ENVIRONMENT

text:
The robot environment consists of a Franka Emika Panda robot operating over a bin which contains various objects. The robot’s observations are 64×64×3 RGB images, and its action space consists of 3 DOF delta position control of the end effector with action magnitudes in the range [-10cm,10cm].
A.3.2 DATA
The data used for finetuning the model used in robot experiments was taken directly from Wu et al. (2021a). This data consists of different viewpoint, lighting conditions, and target objects than what is used in our evaluation.

heading: A.3.3 EVALUATION DETAILS

text:
During evaluation, the agent is specified to complete an object pushing task by a goal image. The agent has 50 timesteps to complete the task, and a trial is measured as successful if the majority of the object overlaps with its target position at some point in the episode. Each method is evaluated over 18 trials, of which 6 consist of objects in an “office” setting, 6 consist of objects in a “kitchen” setting, and 6 consist of objects in a “cleaning” setting (See Figure 9).
To execute the task, the agent performs visual model predictive control using the cross entropy method (CEM). Specifically, the agent takes in 1 frame, and predicts trajectories of 10 time-steps for 200 different sampled action sequences. Trajectories are ranked according to their negative mean squared error to the goal image, averaged across all 10 time-steps. The action distribution refits to the top 20 actions, and repeats for 3 iterations of CEM. Afterwards the best sequence of 10 actions is stepped in the environment in an open loop faction. The process repeats 5 times until the end of the episode.
Ground Truth1s t C
on te
xt F
ra m
e t=
1
Predicted
t= 2
t= 3
t= 4
t= 5
t= 6
t= 7
t= 8
t= 9
t= 10
Ground Truth
Predicted
Fi gu
re 11
:M or
e de
ta ile
d vi
de o
fr om
Fi gu
re 4
w hi
ch ill
us tra
te s
Fi tV
id on
ac tio
nco
nd iti
on ed
R ob
oN et
(D as
ar ie
ta l.,
20 19
). Th
e m
od el
is co
nd iti
on ed
on th
e fir
st tw o fr am es an d is pr ed ic tin g th e ne xt te n fr am es gi ve n th e fu tu re ac tio ns of th e ro bo tic ar m .T he se fig ur es de m on st ra te ho w th e pr ed ic te d m ov em en ts of th e ar m cl os el y fo llo w s th e gr ou nd tru th gi ve n th at th e fu tu re ac tio ns is kn ow n. Th e m od el al so pr ed ic ts de ta ile d m ov em en ts of th e pu sh ed ob je ct s (v is ib le in th e to p ex am pl e) as w el l as fil lin g in th e pr ev io us ly un se en ba ck gr ou nd w ith so m e ra nd om ob je ct s (l oo k at th e ob je ct th at ap pe ar be hi nd th e ro bo tic ar m in th e bo tto m ex am pl e) .A ls o no tic e th e w ro ng pr ed ic tio ns of ro bo ts fin ge rs in th e bo tto m ex am pl e.
Ground Truth1s t C
on te
xt F
ra m
e t=
1
Predicted
t= 3
t= 5
t= 7
t= 9
t= 11
t= 13
t= 15
t= 17
t= 19
t= 21
t= 23
t= 25
Ground Truth
Predicted
Fi gu
re 12
:M or
e de
ta ile
d vi
de o
fr om
Fi gu
re 5
w hi
ch ill
us tr
at es
Fi tV
id on
K IT
T Id
at as
et (G
ei ge
re ta
l., 20
13 b)
.A s
it ca
n be
se en
in th
is fig
ur e,
th e
m od
el ge
ne ra
te s
hi gh
qu al
ity pr
ed ic
tio n
of th
e fu
tu re
in a
dy na
m ic
sc en
e. N
ot e
ho w
in th
e to
p ex
am pl
e Fi
tV id
ke ep
s pr
ed ic
tin g
th e
m ov
em en
to ft
he sh
ad ow
on th
e gr
ou nd
til li
tm ov es ou to ft he fr am e. A ft er th at ,t he m od el ke ep s pu sh in g th e ba ck gr ou nd cl os er in ea ch fr am e, im pl yi ng dr iv in g fo rw ar d. W e no tic ed th at th e qu al ity of pr ed ic tio ns dr op su bs ta nt ia lly fa st er w he n th er e ar e m or e ob je ct s in th e sc en e e. g. th e dr iv in g sc en es in si de a ci ty as ca n be se en in th e bo tto m ex am pl e. T hi s in di ca te s th e m od el st ill fa ils to ge ne ra liz e to m or e co m pl ex sc en es w ith m or e m ov in g su bj ec ts .
Ground Truth1s t C
on te
xt F
ra m
e t=
1
Predicted
t= 3
t= 5
t= 7
t= 9
t= 11
t= 13
t= 15
t= 17
t= 19
t= 21
t= 23
t= 25
Ground Truth
Predicted
Fi gu
re 13
: M
or e
de ta
ile d
vi de
o fr
om Fi
gu re
6 w
hi ch
ill us
tr at
es Fi
tV id
on H
um an
3. 6M
(I on
es cu
et al
., 20
14 ).
T hi
s fig
ur e
de m
on st
ra te
s ex
tr em
el y
de ta
ile d
an d
hu m
an -l
ik e
m ot
io ns
pr ed
ic te
d by
Fi tV
id ,c
on di
tio ne
d on
th e
gi ve
n co
nt ex
tf ra
m es
.H ow
ev er
,o n
cl os
er in
sp ec
tio n,
it ca
n be
se en
th at
th e
hu m
an su
bj ec
ti n
th e
vi de o is ch an gi ng ,f ro m th e te st su bj ec tt o a tr ai ni ng su bj ec t. T hi s is pa rt ic ul ar ly ev id en tf ro m th e cl ot hs .T hi s ph en om en a in di ca te s th at ,a lth ou gh Fi tV id is ca pa bl e of ge ne ra liz in g to th e fr am es ou to ft ra in in g di st ri bu tio n, ho w ev er ,i tm or ph s th e hu m an su bj ec ti nt o a fa m ili ar on e fr om th e tr ai ni ng se ta nd th en pl ay s th e vi de o fr om th e m em or y.
Ground TruthC on
te xt
F ra
m e
t= 1 Predicted t=
3 t=
5 t=
7 t=
9 t=
11 t=
13 t=
15
Ground Truth
Predicted
Fi gu
re 14
:M or
e de
ta ile
d vi
de o
fr om
Fi gu
re 8
w hi
ch ill
us tr
at es
Fi tV
id on
B A
IR ro
bo tp
us hi
ng da
ta se
t( E
be rt
et al
., 20
17 )w
ith no
ac tio
ns .T
he m
od el
is co
nd iti
on ed
on ly
on th
e fir
st fr
am e
an d
is pr
ed ic
tin g
th e
ne xt
16 fr
am es
.G iv
en th
at th
e fu
tu re
ac tio
ns of
th e
ro bo
tic ar
m is
un kn
ow n,
th e
pr ed
ic tio
n ca
n di
ve rg
e su
bs ta
nt ia lly fr om th e gr ou nd tr ut h vi de o. H ow ev er ,t he m od el pr ed ic ts m ov em en ts fo rt he ob je ct s w he ne ve rt he ar m pu sh es th e ob je ct in an im ag in ar y sc en ar io .I ta ls o fil ls th e ba ck gr ou nd w ith ra nd om ob je ct s.

title: FITVID: HIGH-CAPACITY PIXEL-LEVEL VIDEO PREDICTION