Disentangling Factors of Variations Using Few Labels
Learning disentangled representations is considered a cornerstone problem in representation learning. Recently, Locatello et al. (2019) demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to a limited amount of supervision, for example through manual labeling of (some) factors of variation in a few training examples. In this paper, we investigate the impact of such supervision on state-of-the-art disentanglement methods and perform a large scale study, training over 52000 models under well-defined and reproducible experimental conditions. We observe that a small number of labeled examples (0.01--0.5% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-art unsupervised models. Further, we investigate the benefit of incorporating supervision into the training process. Overall, we empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.
After rebuttal edit: No clarifications were made, so I keep my score as is. ------------------------------------------------------ Claims: Explicitly requiring a small number of labels allows for successful learning of disentangled features by either using them as a validation set for hyper parameter tuning, or using them as a supervised loss. Decision: Reject. This paper needs a substantial rewrite to make clear what specific contributions are from the multitude of experiments run in this study. As is, the two contributions stated in the introduction are both obvious and not particularly significant -- that having some labels of the type of disentanglement desired helps when used as a validation set and as a small number of labels for learning a disentangled representation space. There are no obviously stated conclusions about which types of labels are better than others (4.2). Section 3.2 seems to have some interesting findings that small scale supervision can help significantly and fine-grained labeling is not necessarily needed, but I don*t understand why that finding is presented there when Fig. 4 seems to perform a similar experiment on types of labels with no conclusion based on its results. Conclusion sentence of 4.3 is hard to decipher, but I assume is just saying S^2/S beats U/S even when S^2/S is subject to noisy labels. Overall, I find it very difficult to absorb the huge amount of results and find the analysis not well presented.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Disentangling Factors of Variations Using Few Labels
Learning disentangled representations is considered a cornerstone problem in representation learning. Recently, Locatello et al. (2019) demonstrated that unsupervised disentanglement learning without inductive biases is theoretically impossible and that existing inductive biases and unsupervised methods do not allow to consistently learn disentangled representations. However, in many practical settings, one might have access to a limited amount of supervision, for example through manual labeling of (some) factors of variation in a few training examples. In this paper, we investigate the impact of such supervision on state-of-the-art disentanglement methods and perform a large scale study, training over 52000 models under well-defined and reproducible experimental conditions. We observe that a small number of labeled examples (0.01--0.5% of the data set), with potentially imprecise and incomplete labels, is sufficient to perform model selection on state-of-the-art unsupervised models. Further, we investigate the benefit of incorporating supervision into the training process. Overall, we empirically validate that with little and imprecise supervision it is possible to reliably learn disentangled representations.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



After rebuttal edit: No clarifications were made, so I keep my score as is. ------------------------------------------------------ Claims: Explicitly requiring a small number of labels allows for successful learning of disentangled features by either using them as a validation set for hyper parameter tuning, or using them as a supervised loss. Decision: Reject. This paper needs a substantial rewrite to make clear what specific contributions are from the multitude of experiments run in this study. As is, the two contributions stated in the introduction are both obvious and not particularly significant -- that having some labels of the type of disentanglement desired helps when used as a validation set and as a small number of labels for learning a disentangled representation space. There are no obviously stated conclusions about which types of labels are better than others (4.2). Section 3.2 seems to have some interesting findings that small scale supervision can help significantly and fine-grained labeling is not necessarily needed, but I don*t understand why that finding is presented there when Fig. 4 seems to perform a similar experiment on types of labels with no conclusion based on its results. Conclusion sentence of 4.3 is hard to decipher, but I assume is just saying S^2/S beats U/S even when S^2/S is subject to noisy labels. Overall, I find it very difficult to absorb the huge amount of results and find the analysis not well presented.
assistant
After rebuttal edit: No clarifications were made, so I keep my score as is. ------------------------------------------------------ Claims: Explicitly requiring a small number of labels allows for successful learning of disentangled features by either using them as a validation set for hyper parameter tuning, or using them as a supervised loss. Decision: Reject. This paper needs a substantial rewrite to make clear what specific contributions are from the multitude of experiments run in this study. As is, the two contributions stated in the introduction are both obvious and not particularly significant -- that having some labels of the type of disentanglement desired helps when used as a validation set and as a small number of labels for learning a disentangled representation space. There are no obviously stated conclusions about which types of labels are better than others (4.2). Section 3.2 seems to have some interesting findings that small scale supervision can help significantly and fine-grained labeling is not necessarily needed, but I don*t understand why that finding is presented there when Fig. 4 seems to perform a similar experiment on types of labels with no conclusion based on its results. Conclusion sentence of 4.3 is hard to decipher, but I assume is just saying S^2/S beats U/S even when S^2/S is subject to noisy labels. Overall, I find it very difficult to absorb the huge amount of results and find the analysis not well presented.