TIME-AGNOSTIC PREDICTION: PREDICTING PREDICTABLE VIDEO FRAMES
Prediction is arguably one of the most basic functions of an intelligent system. In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult. However, most phenomena naturally pass through relatively predictable bottlenecks-while we cannot predict the precise trajectory of a robot arm between being at rest and holding an object up, we can be certain that it must have picked the object up. To exploit this, we decouple visual prediction from a rigid notion of time. While conventional approaches predict frames at regularly spaced temporal intervals, our time-agnostic predictors (TAP) are not tied to specific times so that they may instead discover predictable "bottleneck" frames no matter when they occur. We evaluate our approach for future and intermediate frame prediction across three robotic manipulation tasks. Our predictions are not only of higher visual quality, but also correspond to coherent semantic subgoals in temporally extended tasks.
The authors present a method on prediction of frames in a video, with the key contribution being that the target prediction is floating, resolved by a minimum on the error of prediction. The authors show the merits of the approach on a synthetic benchmark of object manipulation with a robotic arm. 
Quality: this paper appears to contain a lot of work, and in general is of high quality. 
Clarity: some sections of the paper were harder to digest, but overall the quality of the writing is good and the authors have made efforts to present examples and diagrams where appropriate. Fig 1, especially helps one get a quick understanding of the concept of a `bottleneck` state. 
Originality: To the extent of my knowledge, this work is novel. It proposes a new loss function, which is an interesting direction to explore.
Significance: I would say this work is significant. There appears to be a significant improvement in the visual quality of predictions. In most cases, the L1 error metric does not show such a huge improvement, but the visual difference is remarkable, so this goes to show that the L1 metric is perhaps not good enough at this point. 
Overall, I think this work is significant and I would recommend its acceptance for publication at ICLR. There are some drawbacks, but I don’t think they are major or would justify rejection (see comments below). 
I’m curious as to why you called the method in section 3.2 the “Generalized minimum”? It feels more like a weighted (or preference weighted) minimum to me and confused me a few times as I was reading the paper (GENerative? GENeralized? what’s general about it?). Just a comment.
What results does figure 4 present? Are they only for the grasping sequence? Please specify.  
In connection with the previous comment, I think the results would be more readable if the match-steps were normalized for each sequence (at least for Figure 4). There would be a clearer mapping between fixT methods and the normalized matching step (e.g., we would expect fix0.75 to achieve a matching step of 0.75 instead of 6 / ? ).
Section 4, Intermediate prediction. The statement “the genmin w(t) preference is bell-shaped” is vague. Do you mean a Gaussian? If so, you should say “a Gaussian centered at T/2 and tuned so that …”
Section 4, Bottleneck discovery frequency. I am not entirely convinced by the measuring of bottleneck states. You say that a distance is computed between the predicted object position and the ground-truth object position. If a model were to output exactly the same frame as given in context, would the distance be zero? If so, doesn’t that mean that a model who predicts a non-bottleneck state before or after the robotic arm moves the pieces is estimated to have a very good bottleneck prediction frequency? I found this part of the paper the hardest to follow and the least convincing. Perhaps some intermediate results could help prospective readers understand better and be convinced of the protocol’s merits.
Typos:
Appendix E, 2nd paragraph, first sentence: “... generate an bidirectional state” --> “generate A bidirectional state”