Generative models are important tools to capture and investigate the properties of complex empirical data. Recent developments such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) use two very similar, but reverse, deep convolutional architectures, one to generate and one to extract information from data. Does learning the parameters of both architectures obey the same rules? We exploit the causality principle of independence of mechanisms to quantify how the weights of successive layers adapt to each other. Using the recently introduced Spectral Independence Criterion, we quantify the dependencies between the kernels of successive convolutional layers and show that those are more independent for the generative process than for information extraction, in line with results from the field of causal inference. In addition, our experiments on generation of human faces suggest that more independence between successive layers of generators results in improved performance of these architectures.
This paper examines the nature of convolutional filters in the encoder and a decoder of a VAE, and a generator and a discriminator of a GAN. The authors treat the inputs (X) and outputs (Y) of each filter throughout each step of the convolving process as a time series, which allows them to do a Discrete Time Fourier Transform analysis of the resulting sequences. By comparing the power spectral density of the input and the output, they get a Spectral Dependency Ratio (SDR) ratio that characterises a filter as spectrally independent (neutral), correlating (amplifies certain frequencies), or anti-correlating (dampens frequencies). This analysis is performed in the context of the Independence of Cause and Mechanism (ICM) framework. The authors claim that their analysis demonstrates a different characterisation of the inference/discriminator and generative networks in VAE and GAN, whereby the former are anti-causal and the latter are causal in line with the ICM framework. They also claim that this analysis can be used to improve the performance of the models.
Pros:
-- SDR characterisation of the convolutional filters is interesting
-- The authors show that filters with different characteristics are responsible for different aspects of image modelling
Cons:
-- The authors do not actually demonstrate how their analysis can be used to improve VAEs or GANs
-- Their proposed SDR analysis does not actually find much difference between the generator and the discriminator of the GAN 
-- The clarity of the writing could be improved (e.g. the discussion in section 3.1 seems inaccurate in the current form). Grammatical and spelling mistake are frequent. More background information could be helpful in section 2.2. All figures (but in particular Figure 3) need more informative captions
-- The authors talk a lot about disentangling in the introduction, but this does not seem to be followed up in the rest of the text. Furthermore, they are missing a reference to beta-VAE (Higgins et al, 2017) when discussing VAE-based approaches to disentangled factor learning
In summary, the paper is not ready for publication in its current form. The authors are advised to use the insights from their proposed SDR analysis to demonstrate quantifiable improvements the VAEs/GANs.