Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple autoregressive video generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics on popular benchmark datasets, for which they produce continuations of high fidelity and realism. We also present results from training our models on Kinetics, a large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While modeling these phenomena consistently remains elusive, we hope that our results, which include occasional realistic continuations encourage further research on comparatively complex, large scale datasets such as Kinetics.
This paper presents an approach for scalable autoregressive models for video synthesis. Key to the approach is a form of 3D (2 space and one time) self-attention that operates in a softly local manner (through a bias on the attention weights that makes them tend to prefer nearby connections), and also limits its field of view to a specific 3D sub-"block" of video at each layer for scalability. They also propose a clever ordering for autoregressive synthesis of the video subsampling spatially and temporally to generate multiple slices that are synthesized autoregressively one after the other. Each of these ideas is individually close to ideas proposed elsewhere before in other forms, as the authors themselves acknowledge [Vaswani et al 2017, Parmar et al 2018, Parikh et al 2016, Menick et al 2019], but this paper does the important engineering work of selecting and combining these ideas in this specific video synthesis problem setting.
Results on standard datasets for video generation match up to and/or surpass prior methods, in line with prior work on autoregressive image generation that has been shown to do very well on similar metrics (perplexity and FID). What is perhaps more interesting is that this paper presents initial promising results for open-world Youtube video settings (Kinetics dataset) that have not been evaluated systematically in any prior work in this area, to my knowledge.
The downsides of this paper are largely common to this method class (autoregressive generative models): training time (one of their models is "trained in parallel on 128 TPU v3 instances for 1M steps"), inference time (four short 64x64 video clips of 30 frames take 8 mins to generate on a Tesla V100), and model sizes (373M parameters for the Kinetics model). However, this does not take away from the contributions made here, that make it possible at all to train an autoregressive model of this size. 
On the experiments, some questions, comments, and suggestions that the authors might consider addressing:
- How well do methods like SVG, SAVP, SV2P do on Kinetics, for comparison? It would be still more interesting if those models were scaled to have similar sizes to the large model in this work. While these methods have never been evaluated before on such unconstrained data, it is not clearly established that they do not work at all.
- To what extent does the blocking help, and when does it breaks down? e.g. how many layers/how large do blocks have to be for the idea of using different block sizes to suffice for smooth video synthesis? What happens when the blocking idea is not used at all?
- Other choices that aren't ablated in experiments: the choice of a local preference using the bias term in attention, the Transformer-style multi-attention heads. I do understand that these models are expensive to train and evaluate, but perhaps a smaller dataset might still suffice to demonstrate the value of these choices.
- Why is the proposed approach evaluated only on video prediction? Could it not be used for video generation without conditioning or with class conditioning?
- It is surprising to me that the perplexity of Kinetics models is lower than BAIR. Is there a reasonable explanation?
Writing and presentation are good for the most part, despite the main paper being dense with details and multiple fairly involved ideas. I particularly enjoyed parts of related work, the illustration of slicing in Fig 1, and the illustrative examples in Fig 3. 
I would suggest however, that the paper might benefit from placing Sec 3.2 which describes the framework, before Sec 3.1. Fig 1 also belongs closer to Sec 3.2 anyway.
There are also terms/phrases I don't understand despite being reasonably familiar with the field like "positional embbeddings" (Sec 3.2). I also don't understand the need for "one-hot encoding of the discretized pixel intensities" (in that same paragraph). As a more minor comment, a footnote 1 before Eq 1 declares that capital letters denote matrices right before using capital letters to denote constants (T, H, W etc.).