We propose the factorized action variational autoencoder (FAVAE), a state-of-theart generative model for learning disentangled and interpretable representations from sequential data via the information bottleneck without supervision. The purpose of disentangled representation learning is to obtain interpretable and transferable representations from data. We focused on the disentangled representation of sequential data since there is a wide range of potential applications if disentanglement representation is extended to sequential data such as video, speech, and stock price data. Sequential data is characterized by dynamic factors and static factors: dynamic factors are time dependent, and static factors are independent of time. Previous works succeed in disentangling static factors and dynamic factors by explicit modeling the priors of latent variables to distinguish between static and dynamic factors. However, this model cannot disentangle representations between dynamic factors, such as disentangling "picking" and "throwing" in robotic tasks. In this paper, we propose new model that can disentangle multiple dynamic factors. Since our method does not require modeling priors, it is capable of disentangling "between" dynamic factors. In experiments, we show that FAVAE can extract the disentangled dynamic factors.
This paper presents a new approach to learning disentangled representations of sequential data. The model, FAVAE, is based on the information bottleneck framework (Alemi et al, 2016; Achille et al, 2016) and extends the recent beta-VAE (Higgins et al, 2017) and CCI-VAE (Burgess et al, 2017) work by changing the encoder/decoder to a Quasi-Recurrent Neural Network (QRNN) and adding multiple latents through a ladder VAE approach (Zhao et al, 2017). The authors demonstrate that their approach is able to learn a more disentangled representation than the limited set of baselines on three toy datasets.
The authors address a very important problem, since the ability to learn disentangled representations of sequential data can have far reaching consequences, as the authors rightfully point out in their introduction. I also like the approach that the authors are taking, which appears to be very general and does not seem to involve the need to have any domain knowledge. However, the paper could be improved by clarifying certain key parts and extending the experimental section, which is currently not quite as convincing as I would have hoped.
The summary of my concerns is the following: 
I would like to see more baseline comparisons 1) to an FAVAE with a different recurrent encoder/decoder; 2) to at least one other approach to disentangled representation learning on sequences; 3) an FAVAE without the capacity increase. I would also like to see all the baselines run on a non toy dataset of video data. Finally, I would like to see an expanded discussion of what the different latent variables at the different levels of the ladder architecture are learning. I recommend that the authors remove the MIE metric and shorten Section 3 to make space for the expanded experiments section.
I do hope that the authors are able to address my concerns, because their method has a lot of potential and I am excited to see where they take it during the rebuttal period. Please see the more detailed comments below:
1) Section 4.2 should be expanded to include a more detailed description of QRNN. This is one of the key modifications of FAVAE compared to CCI-VAE or beta-VAE, and it is currently not clear how QRNN works unless one reads the original paper referenced. The current paper needs to be self contained. It would also be nice to get a better understanding why QRNN was chosen over an LSTM or a GRU. It would be useful to see the results of the baseline experiments with an LSTM compared to the equivalent QRNN-based version of FAVAE.
2) Why do the authors introduce the new MIE metric? The reported results do not show a significant problem with the MIG metric, and the need for the new MIE metric is not well motivated. If the authors insist on introducing this new metric, it is important to demonstrate cases where MIG fails and MIE performs better. Otherwise I would advise removing the new metric and using the space to expand section 4.2 instead. 
Another point on the metric, Eq. 16 seems to be missing a term that goes over latents z_j. I assume there should be either a max_j or a mean_j in front of the I(z_j; v_k) term in the first part of Eq. 16?
3) The related work section should be re-worded. Currently it reads as if the other approaches do not optimise a loss function at all. It would also be good to include one of the mentioned models as a baseline, and to run both FAVAE and one of the previously introduced models for disentangled sequential representation learning on a more complex dataset of video data.
4) In section 7.1, it would be good to expand the discussion of how latent traversal plots are obtained. In particular, I do not understand how the different latent variables in the ladder architecture of FAVAE are traversed. In general, it would also be nice to expand the discussion of what the different latents at the different ladder stages learn, and how the number of ladder stages affects the nature of learnt representations.
5) In terms of the baselines, it would be good to see the full ablation study. The way I see it, FAVAE has three modifications on the standard variational approaches: 1) the use of a recurrent encoder/decoder (R); 2) the use of the ladder architecture (L); and 3) the use of the capacity constraint objective (C). Currently the authors show the results of the R--, R-C, and RLC conditions. I would also like to see the results of the RL- condition (where beta=1 and C=0).
6) In terms of the results presented in Tbl. 1, it would be nice to include the hyperparameters that the authors have swept over in the appendix, as well as a table of the architecture parameters and the best hyperparameters found for the models presented in the Experiments section. In Tbl.1 it is currently unclear how many seeds the authors used to calculate the standard deviations. The units of the standard deviations presented are also not clear. Finally, it is not clear whether the differences presented in the table are significant.
7) It would be useful to include some details of the 2D Wavy Reaching dataset in the main text, even if it is just listing the nature of the 5 factors.
8) It would be useful to expand the section that talks about the different settings of C explored (page 7, paragraph 2). Currently the point that the authors are trying to make in that paragraph and Fig. 4 is not clear. I would also recommend having beta in Fig. 4 on linear rather than log scale, as the log scale seems to be somewhat confusing.

Minor points:
-- Page 2/paragraph 2: used disentangle -> used to disentangle
-- P4/p5: FAVAE is disentangled representation -> is for(?) disentangled representation
-- P6/p1: the priors time dependency -> the priors' time dependency
-- P7/p1: scores for2D -> scores for 2D
-- P7/p4: MIE score (gray curve) -> MIE score (green(?) curve)