Much recent research has been devoted to video prediction and generation, yet a lot of previous work has been focused on short-scale time horizons. The hierarchical video prediction method by Villegas et al. (2017) is an example of a state of the art method for long term video prediction. However, their method has limited applicability in practical settings as it requires a ground truth pose (e.g., poses of joints of a human) at training time. This paper presents a long term hierarchical video prediction model that does not have such a restriction. We show that the network learns its own higher level structure (e.g., pose-equivalent hidden variables) that works better in cases where the ground truth pose does not fully capture all of the information needed to predict the next frame. This method gives sharper results than other video prediction methods which do not require a ground truth pose, and its efficiency is shown on the Humans 3.6M and Robot Pushing datasets.
The paper presents a method for hierarchical future frame prediction in monocular videos. It builds upon the recent method of Villegas et al. 2017, which generates future RGB frames in two stages: in the first stage, it predicts a human body pose sequence, then it conditions on the pose sequence to predict RGB content, using an image analogy network. This current paper, does not constrain the first stage (high level) prediction to be human poses, but instead it can be any high level representation. Thus, the method does not require human annotations.
The method has the following two sub-networks:
1) An image encoder, that given an RGB image, predicts a deep feature encoding. 
2) An LSTM predictor, that conditioned on the last observed frame's encoding,  predicts future high level structure p_t. Once enough frames are generated though, it conditions on its own predictions. 
3) A visual analogy network (VAN), that given predicted high level structure p_t, it predicts the pixel image I_t, by applying the transformation from the first to tth frame, as computed by the vector subtraction of the corresponding high level encodings (2nd equation of the paper). VAN is trained to preserve parallelogram relationships in the joint RGB image and high level structure embedding.
The authors experiment with  many different neural network connectivities, e.g., not constraining the predicted high level structure to match the encoder's outputs, constraining the predicted high level structure to match the encoder's output (EPEV), and training together the VAN  and predictor so that VAN can tolerate mistakes of the predictor. Results are shown in H3.6m and the pushobject datasets, and are compared against the method of Villegas et all (INDIVIDUAL). The conclusion seems to be that not constraining the predicted high level structure to match the encoder’s output, but biasing the encoder’s output in the observed frames to represent ground-truth pose information, gives the best results. 
Pros
1) Interesting alternative training schemes are tested
Cons:
1)Numerous English mistakes, e.g.,  ''an intelligent agents", ''we explore ways generate" etc.
2) Equations are not numbered (and thus is hard to refer to them.) E.g., i do not understand the first equation, shouldn’t it be that e_{t-1} is always fixed and equal to the encoding of the last observed (not predicted) frame? Then the subscript cannot be t-1.
3) In H3.6M, the results are only qualitative. The conclusions from the paper are uncertain, partly  due to the difficulty of evaluating the video prediction results.
Given the difficulty of assessing the experimental results quantitatively (one possibility to do so is asking a set of people of which one they think is the most plausible video completion), and given the limited novelty of the paper, though interesting alternative architectures are tried out, it may not be suitable to be part of  ICLR proceedings as a conference paper.