A FRAMEWORK FOR THE QUANTITATIVE EVALUATION OF DISENTANGLED REPRESENTATIONS
Recent AI research has emphasised the importance of learning disentangled representations of the explanatory factors behind data. Despite the growing interest in models which can learn such representations, visual inspection remains the standard evaluation metric. While various desiderata have been implied in recent definitions, it is currently unclear what exactly makes one disentangled representation better than another. In this work we propose a framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available. Three criteria are explicitly defined and quantified to elucidate the quality of learnt representations and thus compare models on an equal basis. To illustrate the appropriateness of the framework, we employ it to compare quantitatively the representations learned by recent state-of-the-art models.
The paper addresses the problem of devising a quantitative benchmark to evaluate the capability of algorithms to disentangle factors of variation in the data. 
*Quality* 
The problem addressed is surely relevant in general terms. However, the contributed framework did not account for previously proposed metrics (such as equivariance, invariance and equivalence). Within the experimental results, only two methods are considered: although Info-GAN is a reliable competitor, PCA seems a little too basic to compete against. The choice of using noise-free data only is a limiting constraint (in [Chen et al. 2016], Info-GAN is applied to real-world data). 
Finally, in order to corroborate the quantitative results, authors should have reported some visual experiments in order to assess whether a change in c_j really correspond to a change in the corresponding factor of variation z_i according to the learnt monomial matrix.
*Clarity*
The explanation of the theoretical framework is not clear. In fact, Figure 1 is straight in identifying disentanglement and completeness as a deviation from an ideal bijective mapping. But, then, the authors missed to clarify how the definitions of D_i and C_j translate this requirement into math. 
Also, the criterion of informativeness of Section 2 is split into two sub-criteria in Section 3.3, namely test set NRMSE and Zero-Shot NRMSE: such shift needs to be smoothed and better explained, possibly introducing it in Section 2.
*Originality*
The paper does not allow to judge whether the three proposed criteria are original or not with respect to the previously proposed ones of [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015]. 
*Significance*
The significance of the proposed evaluation framework is not fully clear. The initial assumption of considering factors of variations related to graphics-generated data undermines the relevance of the work. Actually, authors only consider synthetic (noise-free) data belonging to one class only, thus not including the factors of variations related to noise and/or different classes.
PROS: 
The problem faced by the authors is interesting
CONS:
The criteria of disentanglement, informativeness & completeness are not fully clear as they are presented.
The proposed criteria are not compared with previously proposed ones - equivariance, invariance and equivalence [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015]. Thus, it is not possible to elicit from the paper to which extent they are novel or how they are related..
The dataset considered is noise-free and considers one class only. Thus, several factors of variation are excluded a priori and this undermines the significance of the analysis.
The experimental evaluation only considers two methods, comparing Info-GAN, a state-of-the-art method, with a very basic PCA.
**FINAL EVALUATION**
The reviewer rates this paper with a weak reject due to the following points.
1) The novel criteria are not compared with existing ones [Goodfellow et al. 2009, Lenc & Vedaldi 2015, Cohen & Welling 2014, Jayaraman & Grauman 2015].
2) There are two flaws in the experimental validation:
	2.1) The number of methods in comparison (InfoGAN and PCA) is limited.
	2.2) A synthetic dataset is only considered.
The reviewer is favorable in rising the rating towards acceptance if points 1 and 2 will be fixed. 
**EVALUATION AFTER AUTHORS' REBUTTAL**
The reviewer has read the responses provided by the authors during the rebuttal period. In particular, with respect to the highlighted points 1 and 2, point 1 has been thoroughly answered and the novelty with respect previous work is now clearly stated in the paper. Despite the same level of clarification has not been reached for what concerns point 2, the proposed framework (although still limited in relevance due to the lack of more realistic settings) can be useful for the community as a benchmark to verify the level of disentanglement than newly proposed deep architectures can achieve. Finally, by also taking into account the positive evaluation provided by the fellow reviewers, the rating of the paper has been risen towards acceptance.