Recent advancements in unsupervised disentangled representation learning focus on extending the variational autoencoder (VAE) with an augmented objective function to balance the trade-off between disentanglement and reconstruction. We propose Sequential Residual Variational Autoencoder (SR-VAE) that defines a "Residual learning" mechanism as the training regime instead of the augmented objective function. Our proposed solution deploys two important ideas in a single framework: (1) learning from the residual between the input data and the accumulated reconstruction of sequentially added latent variables; (2) decomposing the reconstruction into decoder output and a residual term. This formulation encourages the disentanglement in the latent space by inducing explicit dependency structure, and reduces the bottleneck of VAE by adding the residual term to facilitate reconstruction. More importantly, SR-VAE eliminates the hyperparameter tuning, a crucial step for the prior state-of-the-art performance using the objective function augmentation approach. We demonstrate both qualitatively and quantitatively that SR-VAE improves the state-of-the-art unsupervised disentangled representation learning on a variety of complex datasets.
Overview:
Authors introduce a new VAE-based method for learning disentangled representations.
The main idea is to apply a “residual learning mechanism”, which resembles an autoregressive model, but here conditioning between sequential steps is done in both latent and input spaces. Namely, for each input, the method involves making several (one per *each* embedding dimension) sequential passes, where each pass takes the residual between the input and the current accumulated output (i.e. pixel-wise difference between two images) as well as the values of the sampled embeddings so far. 
Authors report quantitative and qualitative evaluation with a set of reasonable baselines (FactorVAE, beta-VAE), and the results seem to be slightly better.
Decision:
The writing quality is not great: there are frequent typos and not-so-precise formulations, and is at times hard to follow (see list below).
The method itself is not really well-motivated: there seems to be no formal justification provided, and the informal one is not very clearly explained in the paper. 
Scalability of the method is clearly an issue. Many applications might require the size of embeddings to have hundreds of dimensions, which would mean that the given method cannot really be applied for non-toy problems. 
With all those considerations in mind, I cannot recommend to accept this paper as is, thus the final rating, “reject”.
Additional comments / typos:
* It would be worth seeing how this approach relates to auto-regressive models.
* p2. “encoder maps … x to a latent representation q” - this sentence is not very strict. 
* p2. “.. is defined as following”: as follows?
* p2. “.. in addition of b”
* p3. “.. this reguralizer encourage”
* p7: “… it reduces the solution space and improve training stability“: is there any sort of theoretical reasoning that could support this?
* The reference format is not very readable, probably better to change to (Names, Year).
<update>
Authors addressed some of my concerns (thanks!), thus I increased my rating slightly.
There is still a large concern wrt to the computational complexity. I kind of understand the argument about the hyperparameter tuning, but it seems not really fair to compare to a bruteforce parameter search (one can probably do some sort of bayesian optimization).
</update>