An object can be described as the combination of primary visual attributes. Disentangling such underlying primitives is the long-term objective of representation learning. It is observed that categories have the natural hierarchical characteristics, i.e. any two objects can share some common primitives in a particular category level while they may possess their unique ones in another. However, previous works usually operate in a flat manner (i.e. in a particular level) to disentangle the representations of objects. Though they may obtain the primitives to constitute objects as the categories in that level, their results are obviously not efficient and complete. In this paper, we propose the hierarchical disentangle network (HDN) to exploit the rich hierarchical characteristics among categories to divide the disentangling process in a coarse-to-fine manner, such that each level only focuses on learning the specific representations and finally the common and unique representations in all levels jointly constitute the raw object. Specifically, HDN is designed based on an encoder-decoder architecture. To simultaneously ensure the disentanglement and interpretability of the encoded representations, a novel hierarchical generative adversarial network (GAN) is elaborately designed. Quantitative and qualitative evaluations on four object datasets validate the effectiveness of our method.
This paper proposes a method for solving the following problem: given an Image I from a labelled dataset with a label hierarchy as a tree of depth L, produce a set of vector representations {F_1, F_2 ... F_L}, such that a) the set can be used to reconstruct I as well as possible and b) each representation in the set only contains information about the label at the corresponding level in the label tree.
While I am not extremely familiar with work in disentangled representation learning, the authors claim is true to my knowledge that most work on disentangling factors does not explicitly take into account hierarchical structure as this work does. Therefore, this work appears novel and interesting to me. I will leave the assessment of the degree of novelty to other reviewers/AC who may be more familiar with the literature.
The approach is also effective, and the authors demonstrate through visualizations and experiments that the proposed model can be trained and accomplishes its objectives reasonably well. My overall decision is to accept this paper, but there are some improvements I'd like to see since I found it difficult to understand in some places. 
- There is repeated use of the term "granularity" in the abstract and Sec. 1 which is undefined. What, according to the authors, is the difference between having a hierarchical structure and multi-granularity? I suggest clarifying what is meant by this, or avoid using the term (used hierarchy instead).
- In Sec. 3.2, it appears that what is meant by R_l is actually the set {R_1, ..., R_L}. This would imply that the R's from different levels are randomly combined, and the number of representations combined is always L. Is this correct? In either case, what happens here should be made much clearer. It took me several readings to arrive at this interpretation.
- It took me a while to infer how the results in Figure 3 are generated. There is a sudden switch in Sec. 4.1 from model training details to its use for semantic translation, which was not explained.
Minor suggestions:
- Please use parenthetical citations throughout the paper where appropriate (use \citep{}) to avoid breaking the flow of reading.
- Pg. 1, last line: "us human" -> "humans"
- Pg. 2, line 1: "with others" -> "to others"
- Pg. 2, line 2: "hierarchy structure" -> "hierarchical structure"
- Pg. 2: "multi-granularity nature" -> "multi-granular nature" or "hierarchical nature"
- Pg. 7, line 2: "an significant" -> "a significant"
- Pg. 7, line 4: "At the last" -> "At the end"
- Pg. 7, "it can be reached" -> "it can be seen that"
- Pg. 7: "Table.4.1 gives the evaluation results" -? "Table 1 gives ..."
- Pg. 7, Please revise: "which is deserved to make more efforts"
- Pg. 7: "to applied" -> "to be applied"
- Pg. 8, line 1: Remove "quite"; typo in "leraning"