Learning to disentangle the hidden factors of variations within a set of observations is a key task for artificial intelligence. We present a unified formulation for class and content disentanglement and use it to illustrate the limitations of current methods. We therefore introduce LORD, a novel method based on Latent Optimization for Representation Disentanglement. We find that latent optimization, along with an asymmetric noise regularization, is superior to amortized inference for achieving disentangled representations. In extensive experiments, our method is shown to achieve better disentanglement performance than both adversarial and non-adversarial methods that use the same level of supervision. We further introduce a clustering-based approach for extending our method for settings that exhibit in-class variation with promising results on the task of domain translation.
Summary: this paper proposes to basically combine class-conditional noisy autoencoding with GLO to achieve disentanglement while having only access to the content labels. They demonstrate that the method achieves impressive empirical results both in terms of disentanglement and a limited experiment on unsupervised domain translation.
Decision: Weak Reject. I find the experimental results in this paper very appealing. However, I am weakly rejecting the paper because of 1) some problematic claims put forth in the paper, which I worry might mislead the reader and 2) lack of clarity in describing the procedure in the unsupervised domain translation setting.
Here are some main comments:
1.  KL-divergence v. asymmetric noise
First, the authors claim that regularizing with KL-divergence leads to posterior collapse. But the particular experimental set up is tested on SmallNORB, which only has a small handful of factors of variation anyway). That KL-divergence “causes” posterior collapse is a claim that must be made very carefully. There are some very specific conditions under which this is known to be true empirically (for example, see the experiments in Burda’s IWAE paper and Hoffman’s DLGM paper), but in general, one should be careful with this claim. Can the authors please walk back on this statement?
Second, it is worth noting that asymmetric noise regularization is itself actually a special case of KL-divergence regularzation. When q(z|x) is forced to have a globally fixed variance, KL-divergence regularization becomes asymmetric noise regularization. 
2. Cost of training
One thing I feel should be made more clear in the paper is the training cost of GLO v. amortized models. How much slower is GLO compared to amortized models? How many iterations do you employ on a given minibatch of data when using GLO? 
3. Ablation study
First, I think the authors should show us the actual visualizations for the amortized models. Without visual inspection, it’s hard to gauge the significance of the numbers in Table 3. 
Second, the authors observed that the amortized models leak class information into the content representation. I find it fascinating that GLO does not. I would like the authors to dig deeper into what exactly is the inductive bias conferred by latent optimization. As of the moment, claim that “this variant is inferior to our fully unamortized model as a result of an inductive bias conferred by latent optimization” is a vacuously true statement since we know that amortized models and unamortized models should in theory have equivalent behavior in the infinite-capacity / oracle optimizer regime. I request that the authors show us the training and test losses (Eq 6 and its decomposition into reconstruction + regularization terms). Inspecting it may shed light on the inductive bias. 
4. Unsupervised Domain Translation
The result looks very good. However, the experimentation is too limited. I recommend that the authors try at least one other dataset.
Furthermore, the description of how to apply LORD to unsupervised domain translation is uncomfortably vague. I am not sure if the provided code and description in the main text allows for reproduction of the UDT experiments. 
If the authors are able to address the above questions and requests, then I am more than happy to raise my score.