Deep generative models can emulate the perceptual properties of complex image datasets, providing a latent representation of the data. However, manipulating such representation to perform meaningful and controllable transformations in the data space remains challenging without some form of supervision. While previous work has focused on exploiting statistical independence to disentangle latent factors, we argue that such requirement can be advantageously relaxed and propose instead a non-statistical framework that relies on identifying a modular organization of the network, based on counterfactual manipulations. Our experiments support that modularity between groups of channels is achieved to a certain degree on a variety of generative models. This allowed the design of targeted interventions on complex image datasets, opening the way to applications such as computationally efficient style transfer and the automated assessment of robustness to contextual changes in pattern recognition systems.
Summary:
    Authors propose a method to analyze a trained generator (a neural network) from a trained generative model for a distribution of images and produce a family of modules (subsets of neurons) that control certain aspects of the image (upon intervention) such that the different modules identified are "distentangled" in the sense of a lot of work in this area attempting to find latent controllable factors.
This paper has many original ideas and substantial novelty. One of the prime ones is to focus on existing generators and identify subsets of neurons "inside" the generator box that can be manipulated producing meaningful changes in the image. All existing works attempt to constrain the latent space in an appropriate way during training so as to produce different latent factors that manipulate different aspects of the image. Authors point out that disentangled latents cannot be statistically independent. This is very important (and motivates them to group neurons in the model that are statistically dependent) and I cannot agree more with the authors on this point.
Transformation of the image is defined to be intrinsically disentangled if the same transformation can be produced in the image by intervening (transforming) only a small subset of neurons.
Authors assume that the map from z (latents) to image space (g_M) and map from neurons space to image space induced by the generator (\tilde{g}_M) are invertible .
Authors define a class of manipulations of certain subsets of neurons through counterfactuals. Setting the latents to be the same (unique) z that gave rise to the image Y, just intervene on a susbet E of neurons and set them to h (from the valid range of values it can take) - the result of this intervention for this unit (z) produces a counterfactual image corresponding to y. Now, if the counterfactuals corresponding to the manifold of images map back into the manifold, then it is called a faithful counterfactual.
Authors show a very interesting result - Transformation is disentangled if and only if it is produced by a faithful counterfactual mapping. There is an additional result relating modularity of a subset of neurons to disentanglement.
Then authors produce a concrete algorithm inspired by faithful counterfactual mapping as follows. The authors take two images and two latents z_1 and z_2 that produced them. Then they identify a specific neuron under z_1 , then swap that neuron with the value of the neuron from z_2 with everything else remaining the same under z_1. Then they look at the difference in the images and average across all channels. Then they average over all pairs z_1 and z_2 - this representative "image with one channel" is the average effect of manipulating that neuron.
Then, using nonnegative matrix factorization on the thresholded versions of the average effects of all neurons in a layer, they cluster neurons and then they actually form modules. So now each module can be manipulated together when two images are taken together and neurons in the modules are swapped from one into the other.
Pros:
  I really like the key idea of this paper - counterfactually manipulating neurons using its values from that of some other image and observing differences and clustering them to find similar neurons which could be manipulated as a bunch. Some of the results (although seems handpicked) seem pretty good given this is the first work to group neurons inside a pre-trained generator. I have not seen any work on "intrinsic disentanglement" before. This also has additional implications for people interested in GANs. The fact that pre-trained GANs can lead to counterfactually "realistic" images when "concepts" (clustered neurons of this method)  are swapped shows that GANs really do learn something non-trivial about images. 
I highly recommend this paper for acceptance. However I would like the authors to address some of my concerns. 
Cons:
a)  Typo in Proposition 2 - "then and transformation.." - and should not be there.
b) v_0 is used in some cases - h(z) is used in some places and h (being a constant vector) is used in some places to define counterfactual mapping. Its pretty confusing to read some of the proofs. Is it possible to uniformly define it throughout with constant h and then of course only during experiments and hybridization replace h by v(z_2) (the v from image to be swapped with).
c) Section 3.4 - first paragraph - last line. "Influence maps are grouped by similarity to define modules.." - This is vague. Does this refer to the procedure in the third paragraph of the same section? If so this line could be make to point to this more precise description that comes later.
d) So there are many synthetic experiments where the same pair of images is used to produce many counterfactual images by hybridizing under manipulation of different modules (returned by the clustering algorithm)
What will be useful is for one VAE or GAN ,demonstrate the changes on multiple pairs of images. It seems the image pair is handpicked along with the clustered modules. It would be good to fix the modules and illustrate changes for multiple images using the same set of modules.
e) It seems that the clustered influence maps are simplistic - representing face region, hair region and background. However, ideally we would like to find out - makeup or not, gender , hair color, bangs or not - these are various attributes of the celebA dataset that u get annotated.
Is it possible to correlate obtained modules/influence maps with these annotations already available to find finer concepts that are represented by these annotations ? 
One way to find out is - take a module or a neuron - manipulate it by hybridizing and produce a hybrid image from a pair of original images, build a classifier for one of these annotated labels (attributes) on the CelebA dataset .Then do inference on the hybridized image versus the original two images - if the prediction confidence of the attribute classifier changed across original images and also between the hybrid and one of the original- we know that module/neuron is highly correlated with this attribute. One could build a confusion matrix (sort of) between annotated attributes and all elementary (neurons or modules). This could easily tell us if indeed modules/neurons are capturing all attributes we would like.