Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises. In this work, we show that highly-structured semantic hierarchy emerges as variation factors for synthesizing scenes from the generative representations in state-of-the-art GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with a broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent variation factors facilitates semantic scene manipulation 1 .
The paper presents a visually-guided interpretation of activations of the convolution layers in the generator of StyleGAN on four semantic abstractions (Layout, Scene Category, Scene Attributes and Color), which are referred to as the "Variation Factors" and validates/corroborates these interpretations quantitatively using a re-scoring function. The claim of the paper is that there is a hierarchical encoding in the layers of the StyleGAN generator with respect to the aforementioned "Variation Factors".  Figure 3(a) illustrates how these "Variation Factors" emerge in the layers of the StyleGAN generator.
The basic GAN architecture used in this work is that of StyleGAN. However, details on the architecture of this particular GAN are missing, including in the Appendix. How many Convolution layers are present in its generator? Not everyone is aware of StyleGAN architecture -- A better illustration of their architecture in the main paper and its correspondence with the layer levels (bottom, lower, middle, top) is desired, mainly because the paper is built upon this. The dataset used to train the StyleGAN model is not clear either. In Appendix, Table 1 tabulates the training details, but nowhere is it clearly mentioned if the N=500,000 latent codes are sampled from a GAN model that was trained on a mixture of datasets (i.e., bedroom, living rooms, kitchen etc.) or individual datasets. As well, the unit of training time in Table 1 of the Appendix seems to be M; is it Million or Minutes? Both of them seem unrealistic units for training a GAN.
Since the training dataset is not clear, my understanding of the method is that a range of datasets are used to produce the results, especially for the effect where the transition of Semantic Category results is studied. As a first step, StyleGAN model trained on "bedroom" scenes from LSUN dataset is used to randomly sample codes from the learned distribution, which are further passed through the generator to obtain the respective image mappings. Off-the-shelf image classifiers are employed on each of the images to classify them to one of the four "Visual concepts", which is nothing but the aforementioned four semantic abstractions. Here, I would like to encourage the authors to use a consistent terminology -- The four semantic abstractions have been referred to as "Variation Factors" (page 3), "Candidate concepts" (page 6), "Visual concepts"(page 12), "Semantics" (page 8) interchangeably throughout the literature, which is confusing.  Then, 2000 top positive examples and 2000 top negative examples identified by the image classifiers are used to train a linear SVM, i.e., a binary-SVM all the four scene abstractions ("Varying Factors"), and the separation boundary is obtained. I assume the separation boundary is only obtained once, and not after every layer of the generator. Otherwise, it would not make much sense. 
With the separation boundary (in the form of a normal vector) known for each of the four scene semantics, different feature activations are obtained by moving the latent code towards/away from the separation boundary. A scoring function is obtained to quantify (Equation 1) how the corresponding images vary in a particular semantic aspect when the latent code is moved from the separation boundary.  As per the last line of Paragraph 2 on page 4, a ranking of such scores using this function is used to understand the most relevant latent semantics. Does this mean that initially, a large set of semantics is used to observe whether the output of GAN is manipulated by probing each of them? Or are only four scene semantics chosen to begin with? And what happens when there is a tie? And, what value of K makes this metric more accurate? Any lower bound? Please explain. More questions on the effect of lamda later below.
In the next step, the authors sample a latent code from the learned distribution and pass it through
every layer of the GAN generator. The output code y is varied along the boundary of the SVM classifier. This is repeated at every layer of the GAN generator and the same lamda is used to perturb the resulting output code from the separation boundary. The results are visualized in Fig 3(c). The claim here is that with the same perturbation of the resulting codes (lambda=2) at the output of different GAN layers, the change in the visualized output demonstrates what kind of, if any, semantic is being captured by different layers of GAN. This is also claimed to have been validated through the "re-scoring" function. I am not very clear on this. 
I request the following experiment:
1) Within just a single layer (be it bottom, lower, middle or top), how does the output change when the output code of that layer is perturbed in all directions? This is to see the effect (by visualizing) of the range of lamda values on the output at all the layers. Do you discover any changes weakening your claim?
2) I would like to see the visualizations of the latent codes at the separation boundaries, just to see how well the binary-SVM performs and whether or not, non-binary information is lost/unaccounted for.
On Page-5, see the fourth line from the bottom (going up): how do we know the desired output apriori? Are the four semantic abstractions decided based on the desired output? This takes us back to a question I asked earlier.
Moreover, Layout variation is just view-point variation. So I think it will be appropriate to call it "view-Point Variation" rather than "Layout Variation". This is because Layout is associated with spatial arrangement of objects in a scene, with functionality goals. 
One last question I have is: What is so special about StyleGAN that it was used as the guiding architecture in this work? How generalizable is this approach to other kinds of GANs other than PGGAN and BigGAN (or rather, why is this approach relatable to StyleGAN, PGGAN and BigGAN alone)? 
The paper has grammatical errors  (sentences are not well written), typos (ex; "manipulabe" on page-8 which should be "manipulatable") and is not polished. I also suggest the authors to change the title of the paper, which right now, is a bit odd; if you decide to keep it, there should not be a "the" before "Deep" in the title.
All in all, the paper is interesting but lacks persuasiveness. 
I may jump my score if the authors address all the aforementioned questions and concerns convincingly, and work on the presentation.