Learning disentangled representations that correspond to factors of variation in real-world data is critical to interpretable and human-controllable machine learning. Recently, concerns about the viability of learning disentangled representations in a purely unsupervised manner has spurred a shift toward the incorporation of weak supervision. However, there is currently no formalism that identifies when and how weak supervision will guarantee disentanglement. To address this issue, we provide a theoretical framework to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching. We empirically verify the guarantees and limitations of several weak supervision methods (restricted labeling, match-pairing, and rank-pairing), demonstrating the predictive power and usefulness of our theoretical framework.
The paper tries to bring some theoretical foundation to the weakly supervised disentanglement. Overall it is a good contribution, but the message of the paper is not clear.  The authors propose two notions: consistency and restrictiveness, which they don't imply each other. However, the experiment on real data shows that they are highly correlated.   Up until the experiment section, the paper is well written (although a bit verbose). It seems that it is great but unfinished work.
The paper is well written, but in my opinion, there is too much verbosity on page 4-5 on rather trivial definitions consistency and restrictiveness and a big box in the calculus of disentanglement that steals space from the main results.  In my opinion, those sections can be reduced so that other theorem can be covered. In my opinion, the theorem nine should be part of the main text. 
I understand the definition of "Sufficiency for Disentanglement " but it is not clear why it is important. Sure, it is a strong definition that says for any  $\mathcal{H}$ (and not a subset) the algorithm ($\mathcal{A}$) should be able to match the distribution of the observation but why is it a big deal according to the next paragraph?
I don't see any proof that Eq.11 should be between [0,1]. Yes, g is optimal, and if you enter suboptimal values to it, one expects the nominator to be less than dominator. However, g a function that is optimal in expectation, which does not mean for every s value it nominator is less than the denominator. In fact, some of the values in fig 3 are small negatives.
Fig 3 is not explained well: you are showing normalized consistency and restiveness. First of all, what is the dataset you tried this on? Second, why some values are negative?! These are supposed to be between [0,1]. Third, what is the take-home-message of this figure? the first two matrices from left show that the factors are consistent b/c they are almost diagonal. The third one from left shows that the algorithm you used is not restrictive? Then are you suggesting this as a metric of evaluation? I am not sure I understand the first figure from the right.
Overall, the authors perform a significant amount of experiments, but they did a poor job in summarizing the results.
Finally, the authors claim 
                  "...We believe this correlation between consistency and restrictiveness to have been a general source of confusion in the disentanglement literature, causing many to either observe or believe that restricted labeling or share pairing on $S_i$ (which only guarantees consistency) is sufficient for disentangling Si ..." 
Each of those methods should be analyzed separately to ensure that their algorithms do not induce restiveness. I just don't see the natural connection between your figure 4 and this conclusion that you made.
Minor:
Where is the proof for Theorem 1? In the Supp, it starts with Theorem 8, I guess you meant Lemma 8? You need to clean up the Supp so that one can find the proof easily. I suggest restructuring the Supp to less and finally proof of Thorem 1.