Unsupervised Learning of Disentangled Representations from Video
We present a new model DRNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.
This paper presents a neural network architecture and video-based objective function formulation for the disentanglement of pose and content features in each frame. The proposed neural network consists of encoder CNNs and a decoder CNN. The encoder CNNs are trained to decompose frame inputs into contents features (i.e. background, colors of present structures, etc), and pose features (i.e. current configuration of the structures in the video). A decoder CNN is then used to combine the content features of one frame and pose features of a different frame to reconstruct the frame that corresponds to the pose features. This paper presents a new loss for the disentanglement of pose features from input images using a discriminator network (similar to adversarial loss). They train a discriminator network find similarities between features from the pose encoder of images from the same video but different time steps. The pose encoder is then trained to find features that the discriminator cannot distinguish as being similar. At convergence, the pose encoder features will only contain difference between the frames which should reflect the pose change of the person in time. After the encoder-decoder CNN has been trained to identify the content and pose features, an LSTM is trained to take the content features of the last frame and previous pose features to predict the next pose features. The disentangled content and pose features result in high quality video prediction. The experimental results backup the advantage of this method in terms of feature quality for classification tasks, and shows quantitative and qualitative performance boost over the state-of-the-art in video prediction.
Pros:
Novel video-based loss functions for feature disentanglement.
Excellent results on video prediction using the learned pose and content features.
Cons:
Although this paper claims to be an unsupervised method, the proposed objective function actually needs some weak supervision that different videos should have different content, e.g. object instances; otherwise the second term in equation 3 could be negatively affected (classifying images from different videos as different when they may look the same).
The pose LSTM is trained separately from the encoder-decoder CNN. Can the authors comment on why joint end-to-end training was not performed? In fact, similar networks for end-to-end video prediction have been experimented in Oh et al, NIPS 2015. 
Some typos in equations and text.
Should equation 3 be L_{adversarial}(C) instead of L_{adversarial}(D)?
h^t_c in LSTM input could be denoted as h^n where n is the last observed frame index, t makes it seem as the current step in generation.
Lines 204 and 219 point to Fig. 4.3 which is not in the main paper.
Figure 4 should have matching test sequences since it expect us to compare the “left”, “middle”, and “right” results.
Action classification experiments are a little confusing:
It is not clear what features are used to determine action class and how they are used (i.e. concatenated features? Single features and average for each video? Classification on clips?). The paper states: “In either case, we train a two layer classifier network S on top of either hc or hp, with its output predicting the class label y.”
In KTH, the content is not a good indicator about the action that is happening in the video since the same people perform all actions in similar backgrounds. Thus it’s not clear how the content features can be useful for action classification.
Classification results of network trained without pose disentanglement loss can make classification results stronger if previous issues are clarified.
Pose estimation results using the disentangled pose features are strongly recommended that will significantly strengthen the paper.
For the SUNCG dataset (3D object renderings), comparison with the weakly-supervised pose disentangling work by Yang et al, NIPS 2015 is suggested to highlight the performance of the proposed method.
The use of the first term in L_{adversarial}(E_p) is not clear to me, it seems that the first term and second term in this loss are pulling the features in completely opposite directions which, to my understanding, is helping D instead of confusing it. Isn’t confusing D the whole purpose of  L_{adversarial}(E_p) to confuse D?
Title can be more specific to the contribution (adversarial-like loss for content-pose feature disentangling)