We study the problem of building models that disentangle independent factors of variation. Such models encode features that can efficiently be used for classification and to transfer attributes between different images in image synthesis. As data we use a weakly labeled training set, where labels indicate what single factor has changed between two data samples, although the relative value of the change is unknown. This labeling is of particular interest as it may be readily available without annotation costs. We introduce an autoencoder model and train it through constraints on image pairs and triplets. We show the role of feature dimensionality and adversarial training theoretically and experimentally. We formally prove the existence of the reference ambiguity, which is inherently present in the disentangling task when weakly labeled data is used. The numerical value of a factor has different meaning in different reference frames. When the reference depends on other factors, transferring that factor becomes ambiguous. We demonstrate experimentally that the proposed model can successfully transfer attributes on several datasets, but show also cases when the reference ambiguity occurs.
The paper considers the challenges of disentangling factors of variation in images: for example disentangling viewpoint from vehicle type in an image of a car. They identify a well-known problem, which they call "reference ambiguity", and show that in general without further assumptions one cannot tell apart two different factors of variation. 
They then go on to suggest an interesting AE+GAN architecture where the main novelty is the idea of taking triplets such that the first two instances vary in only one factor of variation, while the third instance varies in both from the pair. This is clever and allows them to try and disentangle the variation factors using a joint encoder-decoder architecture working on the triplet.
Pros:
1. Interesting use of constructed triplets.
2. Interesting use of GAN on the artificial instance named x_{3 \oplus 1}
Cons:
1. Lack of clarity: the paper is hard to follow at times. It's not entirely obvious how the theoretical part informs the practical part. See detailed comments below.
2. The theory addresses two widely recognized problems as if they're novel:  "reference ambiguity" and "shortcut problem". The second merely refers to the fact that unconstrained autoencoders will merely memorize the instance. 
3. Some of the architectural choices (the one derived from "shortcut problem") are barely explained or looked into.
Specific comments:
1. An important point regarding the reference ambiguity problem and eq. (2): a general bijective function mixing v and c would not have the two components as independent. The authors could have used this extremely important aspect of the generative process they posit in order to circumvent the problem of ambiguity. In fact, I suspect that this is what allows their method to succeed.
2. I think the intro could be made better if more concrete examples be made earlier on. Specifically the car-type/viewpoint example, along with noting what weak labels mean in that context.
3. In presenting autoencoders it is crucial to note that they are all built around the idea of compression. Otherwise, the perfect latent representation is z=x.
4. I would consider switching the order of sections 2 and 3, so the reader will be better grounded in what this paper is about before reading the related work.
5. In discussing attributes and "valid" features, I found the paper rather vague. An image has many attributes: the glint in the corner of a window, the hue of a leaf. The authors should be much more specific in this discussion and definite explicitly and clearly what they mean when they use these terms.
6. In equation (5), should it be p(v_1,v_2)? Or are v_1 and v_2 assumed to be independent? 
7. Under equation (5), the paper mentions an "autoencoder constraint". Such a constraint is not mentioned up to this point in the paper if I'm not mistaken. 
8. Also under equation (5): is this where the encoder requirements are defined? If so, please be more explicit about it. Also note that you should require c_1 \neq c_2. 
9. In proof of Proposition 1, there is discussion of N_c. N_c was mentioned before but never properly defined; same for R_c and C^-1. These should be part of the proposition statement or defined formally. Currently they are only discussed ad-hoc after equation (5). 
10 .In the proof of Proposition 1, what is f_c^-1 ? It's only defined later in the paper.
11. In general, what promises that f_c^-1 and f_v^-1 are well defined? Are f_c and f_v injective? Why? 
12. Before explaining the training of the model, the task should be defined properly. What is the goal of the training? 
13. In eq. (15) I am missing a term which addresses "the shortcut problem" as defined in the previous page.
14. The weak labels are never properly defined and are discussed in a vague manner. Please define what does that term mean in your context and what were the weak labels in each experiment.
15. In the conclusion, I would edit to say the "our trained model works well on *several* datasets". 
Minor comments:
Please use \citep when appropriate. Instead of "Generative Adversarial Nets Goodfellow et al. (2014)", you should have "Generative Adversarial Nets (Goodfellow et al., 2014)"