Learning disentangled representations is considered a cornerstone problem in representation learning. Recently, Locatello et al. (2019)  demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to a limited amount of supervision, for example through manual labeling of (some) factors of variation in a few training examples. In this paper, we investigate the impact of such supervision on state-of-the-art disentanglement methods and perform a large scale study, training over 52 000 models under well-defined and reproducible experimental conditions. We observe that a small number of labeled examples (0.01-0.5% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-art unsupervised models. Further, we investigate the benefit of incorporating supervision into the training process. Overall, we empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.
This paper considers the challenge of learning disentangled representations---i.e. learning representations of data points x, r(x), that capture the factors of variation in an underlying latent variable z that controls the generating process of x---and studies two approaches for integrated a small number of data points manually labeled with z (or noisier variants thereof): one using these to perform model selection, and another incorporating them into unsupervised representation learning via an additional supervised loss term.  This investigation is motivated by recent results concluding that inductive biases are needed otherwise learning disentangled representations in an unsupervised fashion is impossible.  The paper poses its overall goal as making this injection of inductive biases explicit via a small number (~100 even) of (potentially noisy) labels, and reports on exhaustive experiments on four datasets.
I think that this paper merits acceptance because (a) the motivation of taking a necessity in practice (somehow selecting models / injecting inductive biases) and making it more explicit in the approach is a good one, and because the thorough empirical survey (and simple, but novel, contribution of a new semi-supervised representation learning objective) are likely valuable contributions to this community.
One negative comment overall would be that the results are not that surprising: that is, the fact that using labels either (a) to do model validation or (b) in a semi-supervised fashion would help is not too surprising.  However, I believe in the context of (a) making more explicit a practical (and theoretically) necessary step in the pipeline of learning representations, and (b) contributing a comprehensive empirical study, this is a worthwhile contribution.
Minor notes:
- Fig. 1 isn't the most intuitive- ideally would be better explained for the headlining figure
- 8.57 P100 GPU years is ~= $75k based on a cursory glance at cloud instance pricing at monthly rates... this is a lot to reproduce these experiments...