Deep generative models such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) are important tools to capture and investigate the properties of complex empirical data. However, the complexity of their inner elements makes their functionment challenging to assess and modify. In this respect, these architectures behave as black box models. In order to better understand the function of such networks, we analyze their modularity based on the counterfactual manipulation of their internal variables. Our experiments on the generation of human faces with VAEs and GANs support that modularity between activation maps distributed over channels of generator architectures is achieved to some degree, can be used to better understand how these systems operate and allow meaningful transformations of the generated images without further training.
This paper proposes to examine generative adversarial networks by using counterfactual reasoning. The authors propose to examine modularity through the lens of interventions on the generative networks. After observing that the nodes within the generative network obey a deterministic relationship, they propose a proxy for intervention which takes samples and creates “hybrid” samples by replacing the activation output of one sample with the others. Given the vast number of nodes that exist within a generative network, the authors propose a heuristic for choosing the nodes to perturb. 
I found the underlying premise of this paper to be very strong (identifying modularity in generative networks), however I think there is a substantial amount of work that should go into this paper before acceptance. While the authors begin by working within the framework of causal reasoning there is no mention of what the effect is that they are seeking to measure, i.e. what is the causal estimand here? The influence maps provide an intuitive answer to this, but not one that defines a clear estimand. I would like to see additional evaluation. The evidence provided largely leaves the reader to interpret results subjectively, rather than providing clear evidence. I was also uncomfortable with the selection on hyperparameters (3 clusters). It would be very nice to either have a selection criterion or show the sensitivity of the proposed methodology to other choices. 
Overall, I think this is an interesting idea in a very important area, but one that is not quite ready for publication.
Some editorial comments:
The layout of this paper is slightly strange. After the introduction, the authors introduce the notion of disentanglement and lead with an example from optics. This motivation should either be moved to the introduction or removed. After the definitions the authors jump into a related work section that feels slightly disjointed from the previous section.
I found definition 1 to be abstruse. In addition there are a couple of typos that should be addressed (“consists in a distribution” → “consists of a distribution”). It is non-standard to lead with the latent variables. I think it makes for a much easier narrative to describe the observed variables and structure first, before carrying on to the latent variables. Additionally, I believe you are stating an observation made by Pearl (2001) that after observing the noise variable, relationships become deterministic. This is slightly non-obvious from the wording used (and is also missing the proper reference).
Parens are missing from the following citation:
“generative models encountered in machine learning Besserve et al. (2018).”  → “generative models encountered in machine learning (Besserve et al., 2018).”