REVISITING BATCH NORMALIZATION FOR PRACTICAL DOMAIN ADAPTATION
Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al., 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics from the source domain to the target domain in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives stateof-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.
1 INTRODUCTION
Training a DNN for a new image recognition task is expensive. It requires a large amount of labeled training images that are not easy to obtain. One common practice is to use labeled data from other related source such as a different public dataset, or harvesting images by keywords from a search engine. Because 1) the distributions of the source domains (third party datasets or Internet images) are often different from the target domain (testing images); and 2) DNN is particularly good at capturing dataset bias in its internal representation (Torralba & Efros, 2011), which eventually leads to overfitting, imperfectly paired training and testing sets usually leads to inferior performance.
Known as domain adaptation, the effort to bridge the gap between training and testing data distributions has been discussed several times under the context of deep learning (Tzeng et al., 2014; Long et al., 2015; Tzeng et al., 2015; Ganin & Lempitsky, 2015). To make the connection between the domain of training and the domain of testing, most of these methods require additional optimization steps and extra parameters. Such additional computational burden could greatly complicate the training of a DNN which is already intimidating enough for most people.
In this paper, we propose a simple yet effective approach called AdaBN for batch normalized DNN domain adaptation. We hypothesize that the label related knowledge is stored in the weight matrix of each layer, whereas domain related knowledge is represented by the statistics of the Batch Normalization (BN) (Ioffe & Szegedy, 2015) layer. Therefore, we can easily transfer the trained model to a new domain by modulating the statistics in the BN layer. This approach is straightforward to implement, has zero parameter to tune, and requires minimal computational resources. Moreover, our AdaBN is ready to be extended to more sophisticated scenarios such as multi-source domain adaptation and semi-supervised settings. Fig. 1 illustrates the flowchart of AdaBN. To summarize, our contributions are as follows:
1. We propose a novel domain adaptation technique called Adaptive Batch Normalization (AdaBN). We show that AdaBN can naturally dissociate bias and variance of a dataset, which is ideal for domain adaptation tasks.
2. We validate the effectiveness of our approach on standard benchmarks for both single source and multi-source domain adaptation. Our method outperforms the state-of-the-art methods.
3. We conduct experiments on the cloud detection for remote sensing images to further demonstrate the effectiveness of our approach in practical use.
2 RELATED WORK
Domain transfer in visual recognition tasks has gained increasing attention in recent literature (Beijbom, 2012; Patel et al., 2015). Often referred to as covariate shift (Shimodaira, 2000) or dataset bias (Torralba & Efros, 2011), this problem poses a great challenge to the generalization ability of a learned model. One key component of domain transfer is to model the difference between source and target distributions. In Khosla et al. (2012), the authors assign each dataset with an explicit bias vector, and train one discriminative model to handle multiple classification problems with different bias terms. A more explicit way to compute dataset difference is based on Maximum Mean Discrepancy (MMD) (Gretton et al., 2012). This approach projects each data sample into a Reproducing Kernel Hilbert Space, and then computes the difference of sample means. To reduce dataset discrepancies, many methods are proposed, including sample selections (Huang et al., 2006; Gong et al., 2013), explicit projection learning (Pan et al., 2011; Gopalan et al., 2011; Baktashmotlagh et al., 2013) and principal axes alignment (Fernando et al., 2013; Gong et al., 2012; Aljundi et al., 2015).
All of these methods face the same challenge of constructing the domain transfer function – a highdimensional non-linear function. Due to computational constraints, most of the proposed transfer functions are in the category of simple shallow projections, which are typically composed of kernel transformations and linear mapping functions.
In the field of deep learning, feature transferability across different domains is a tantalizing yet generally unsolved topic (Yosinski et al., 2014; Tommasi et al., 2015). To transfer the learned representations to a new dataset, pre-training plus fine-tuning (Donahue et al., 2014) have become de facto procedures. However, adaptation by fine-tuning is far from perfect. It requires a considerable amount of labeled data from the target domain, and non-negligible computational resources to retrain the whole network.
A series of progress has been made in DNN to facilitate domain transfer. Early works of domain adaptation either focus on reordering fine-tuning samples (Chopra et al., 2013), or regularizing MMD (Gretton et al., 2012) in a shallow network (Ghifary et al., 2014). It is only until recently that the problem is directly attacked under the setting of classification of unlabeled target domain using modern convolutional neural network (CNN) architecture. DDC (Tzeng et al., 2014) used the classical MMD loss to regularize the representation in the last layer of CNN. DAN (Long et al., 2015) further extended the method to multiple kernel MMD and multiple layer adaptation. Besides adapting features using MMD, RTN (Long et al., 2016) also added a gated residual layer for classifier adaptation. RevGrad (Ganin & Lempitsky, 2015) devised a gradient reversal layer to compensate the back-propagated gradients that are domain specific. Recently, by explicitly modeling both private and shared components of the domain representations in the network, Bousmalis et al. (2016) proposed a Domain Separation Network to extract better domain-invariant features.
Another related work is CORAL (Sun et al., 2016). This model focuses on the last layer of CNN. CORAL whitens the data in source domain, and then re-correlates the source domain features to target domain. This operation aligns the second order statistics of source domain and target domain distributions. Surprisingly, such simple approach yields state-of-the-arts results in various text classification and visual recognition tasks. Recently, Deep CORAL (Sun & Saenko, 2016) also extends the method into DNN by incorporating a CORAL loss.
2.1 BATCH NORMALIZATION
In this section, we briefly review Batch Normalization (BN) (Ioffe & Szegedy, 2015) which is closely related to our AdaBN. The BN layer is originally designed to alleviate the issue of internal covariate shifting – a common problem while training a very deep neural network. It first standardizes each feature in a mini-batch, and then learns a common slope and bias for each mini-batch. Formally, given the input to a BN layer X ∈ Rn×p, where n denotes the batch size, and p is the feature dimension, BN layer transforms a feature j ∈ {1 . . . p} into:
x̂j = xj − E[X·j ]√
Var[X·j ] ,
yj = γj x̂j + βj ,
(1)
where xj and yj are the input/output scalars of one neuron response in one data sample; X·j denotes the jth column of the input data; and γj and βj are parameters to be learned. This transformation guarantees that the input distribution of each layer remains unchanged across different mini-batches. For Stochastic Gradient Descent (SGD) optimization, a stable input distribution could greatly facilitate model convergence, leading to much faster training speed for CNN. Moreover, if training data are shuffled at each epoch, the same training sample will be applied with different transformations, or in other words, more comprehensively augmented throughout the training. During the testing phase, the global statistics of all training samples is used to normalize every mini-batch of test data.
Extensive experiments have shown that Batch Normalization significantly reduces the number of iteration to converge, and improves the final performance at the same time. BN layer has become a standard component in recent top-performing CNN architectures, such as deep residual network (He et al., 2016), and Inception V3 (Szegedy et al., 2015).
3 THE MODEL
In Sec. 3.1, we first analyze the domain shift in deep neural network, and reveal two key observations. Then in Sec. 3.2, we introduce our Adaptive Batch Normalization (AdaBN) method based on these observations.
3.1 A PILOT EXPERIMENT
The Batch Normalization (BN) technique is originally proposed to help SGD optimization by aligning the distribution of training data. From this perspective, it is interesting to examine the BN parameters (batch-wise mean and variance) over different dataset at different layers of the network.
In this pilot experiment, we use MXNet implementation (Chen et al., 2016b) of the Inception-BN model (Ioffe & Szegedy, 2015) pre-trained on ImageNet classification task (Russakovsky et al., 2015) as our baseline DNN model. Our image data are drawn from (Bergamo & Torresani, 2010), which contains the same classes of images from both Caltech-256 dataset (Griffin et al., 2007) and Bing image search results. For each mini-batch sampled from one dataset, we concatenate the mean and variance of all neurons from one layer to form a feature vector. Using linear SVM, we can almost perfectly classify whether the mini-batch feature vector is from Caltech-256 or Bing dataset. Fig. 2 visualizes the distributions of mini-batch feature vectors from two datasets in 2D. It is clear that BN statistics from different domains are separated into clusters.
This pilot experiment suggests:
1. Both shallow layers and deep layers of the DNN are influenced by domain shift. Domain adaptation by manipulating the output layer alone is not enough.
2. The statistics of BN layer contain the traits of the data domain.
Both observations motivate us to adapt the representation across different domains by BN layer.
3.2 ADAPTIVE BATCH NORMALIZATION
Given the pre-trained DNN model and a target domain, our Adaptive Batch Normalization algorithm is as follows1:
Algorithm 1 Adaptive Batch Normalization (AdaBN) for neuron j in DNN do
Concatenate neuron responses on all images of target domain t: xj = [. . . , xj(m), . . .] Compute the mean and variance of the target domain: µtj = E(xtj), σtj = √ Var(xtj). end for for neuron j in DNN, testing imagem in target domain do
Compute BN output yj(m) := γj
( xj(m)−µtj ) σtj + βj
end for
The intuition behind our method is straightforward: The standardization of each layer by domain ensures that each layer receives data from a similar distribution, no matter it comes from the source
1In practice we adopt an online algorithm (Donald, 1999) to efficiently estimate the mean and variance.
domain or the target domain. Although modulating statistics in one BN layer by AdaBN is a simple translation and scaling operation, such linear transformation in one layer can achieve a highly nonlinear transformation through the whole deep CNN architecture. Thus, we believe this AdaBN process could approximate the intrinsically non-linear domain transfer function.
For K domain adaptation where K > 2, we standardize each sample by the statistics in its own domain. During training, the statistics are calculated for every mini-batch, the only thing that we need to make sure is that the samples in every mini-batch are from the same domain. For (semi)supervised domain adaptation, we may use the labeled data to fine-tune the weights as well. As a result, our method could fit in all different settings of domain adaptation with minimal effort.
Compared with CORAL (Sun et al., 2016), one natural question is why we transform the neuron responses independently, not decorrelate and then re-correlate the responses together as suggested in Sun et al. (2016). Under certain conditions, decorrelation could improve the performance. However, in CNN, the mini-batch size is usually smaller than the feature dimension, leading to singular covariance matrices that is hard to be inversed. As a result, the covariance matrix is always singular. In addition, decorrelation requires to compute the inverse of the covariance matrix which is computationally intensive, especially if we plan to apply AdaBN to all layers of the network.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of AdaBN on standard domain adaptation datasets, and empirically analyze our AdaBN model. We also evaluation our method on a practical application with remote sensing images.
4.1 EXPERIMENTAL SETTINGS
We first introduce our experiments on two standard datasets: Office (Saenko et al., 2010) and Caltech-Bing (Bergamo & Torresani, 2010).
Office (Saenko et al., 2010) is a standard benchmark for domain adaptation, which is a collection of 4652 images in 31 classes from three different domains: Amazon(A), DSRL(D) and Webcam(W). Similar to (Tzeng et al., 2014; Sun et al., 2016; Long et al., 2015), we evaluate the pairwise domain adaption performance of AdaBN on all six pairs of domains. For the multi-source setting, we evaluate our method on three transfer tasks {A, W} → D, {A, D} →W, {D, W} → A. Caltech-Bing (Bergamo & Torresani, 2010) is a much larger domain adaptation dataset, which contains 30,607 and 121,730 images in 256 categories from two domains Caltech-256(C) and Bing(B). The images in the Bing set are collected from Bing image search engine by keyword search. Apparently Bing data contains noise, and its data distribution is dramatically different from that of Caltech-256.
We compare our approach with a variety of methods, including four shallow methods: SA (Fernando et al., 2013), LSSA (Aljundi et al., 2015), GFK (Gong et al., 2012), CORAL (Sun et al., 2016), and four deep methods: DDC (Tzeng et al., 2014), DAN (Long et al., 2015), RevGrad (Ganin & Lempitsky, 2015), Deep CORAL (Sun & Saenko, 2016). Specifically, GFK models domain shift by integrating an infinite number of subspaces that characterize changes in statistical properties from the source to the target domain. SA, LSSA and CORAL align the source and target subspaces by explicit feature space transformations that would map source distribution into the target one. DDC and DAN are deep learning based methods which maximize domain invariance by adding to AlexNet one or several adaptation layers using MMD. RevGrad incorporates a gradient reversal layer in the deep model to encourage learning domain-invariant features. Deep CORAL extends CORAL to perform end-to-end adaptation in DNN. It should be noted that these deep learning methods have the adaptation layers on top of the output layers of DNNs, which is a sharp contrast to our method that delves into early convolution layers as well with the help of BN layers.
We follow the full protocol (Donahue et al., 2014) for the single source setting; while for multiple sources setting, we use all the samples in the source domains as training data, and use all the samples in the target domain as testing data. We fine-tune the Inception-BN (Ioffe & Szegedy, 2015) model on source domain in each task for 100 epochs. The learning rate is set to 0.01 initially, and then is dropped by a factor 0.1 every 40 epochs. Since the office dataset is quite small, following the
best practice in Long et al. (2015), we freeze the first three groups of Inception modules, and set the learning rate of fourth and fifth group one tenth of the base learning rate to avoid overfitting. For Caltech-Bing dataset, we fine-tune the whole model with the same base learning rate.
4.2 RESULTS
4.2.1 OFFICE DATASET
Our results on Office dataset is reported in Table 1 and Table 2 for single/multi source(s), respectively. Note that the first 5 models of the Table 1 are pre-trained on AlexNet (Krizhevsky et al., 2012) instead of the Inception-BN (Ioffe & Szegedy, 2015) model, due to the lack of publicly available pre-trained Inception BN model in Caffe (Jia et al., 2014). Thus, the relative improvements over the baseline (AlexNet/Inception BN) make more sense than the absolute numbers of each algorithm.
From Table 1, we first notice that the Inception-BN indeed improves over the AlexNet on average, which means that the CNN pre-trained on ImageNet has learned general features, the improvements on ImageNet can be transferred to new tasks. Among the methods based on Inception-BN features, our method improves the most over the baseline. Moreover, since our method is complementary to other methods, we can simply apply CORAL on the top of AdaBN. Not surprisingly, this simple combination exhibits 0.5% increase in performance. This preliminary test reveals further potential of AdaBN if combined with other advanced domain adaptation methods. Finally, we could improve 1.7% over the baseline, and advance the state-of-the-art results for this dataset.
None of the compared methods has reported their performance on multi-source domain adaptation. To demonstrate the capacity of AdaBN under multi-domain settings, we compare it against CORAL, which is the best performing algorithm in the single source setting. The result is reported in Table 2. We find that simply combining two domains does not lead to better performance. The result is generally worse compared to the best performing single domain between the two. This phenomenon suggests that if we cannot properly cope with domain bias, the increase of training samples may be reversely affect to the testing performance. This result confirms the necessity of domain adaptation. In this more challenging setting, AdaBN still outperforms the baseline and CORAL on average. Again, when combined with CORAL, our method demonstrates further improvements. At last, our method archives 2.3% gain over the baseline.
4.2.2 CALTECH-BING DATASET
To further evaluate our method on the large-scale dataset, we show our results on Caltech-Bing Dataset in Table 3. Compared with CORAL, AdaBN achieves better performance, which improves 1.8% over the baseline. Note that all the domain adaptation methods show minor improvements over the baseline in the task C→ B. One of the hypotheses to this relatively small improvement is that the images in Bing dataset are collected from Internet, which are more diverse and noisier (Bergamo & Torresani, 2010). Thus, it is not easy to adapt on the Bing dataset from the relatively clean dataset Caltech-256. Combining CORAL with our method does not offer further improvements. This might be explained by the noise of the Bing dataset and the imbalance of the number of images in the two domains.
4.3 EMPIRICAL ANALYSIS
In this section, we investigate the influence of the number of samples in target domain to the performance and empirically analyze the adaptation effect of different BN layers.
4.3.1 SENSITIVITY TO TARGET DOMAIN SIZE.
Since the key of our method is to calculate the mean and variance of the target domain on different BN layers, it is very natural to ask how many target images is necessary to obtain stable statistics. In this experiment, we randomly select a subset of images in target domain to calculate the statistics and then evaluate the performance on the whole target set. Fig. 3 illustrates the effect of using different number of batches. The results demonstrate that our method can obtain good results when using only a small part of the target examples. It should also be noted that in the extremal case of one batch of target images, our method still achieves better results than the baseline. This is valuable in practical use since a large number of target images are often not available.
4.3.2 ADAPTATION EFFECT FOR DIFFERENT BN LAYERS.
In this experiment, we analyze the effect of adapting on different BN layers with our AdaBN method. According to the structure of Inception-BN network Ioffe & Szegedy (2015), we categorize the BN layers into 9 blocks: 1, 2, 3a, 3b, 4a, 4b, 4c, 5a, 5b. Since the back BN layers are influenced by the outputs of previous BN layers, when adapting a specific block we adapted all the blocks before it. Fig. 4 illustrates the adaptation effect for different BN layers. It shows that adapting BN layers consistently improves the results over the baseline method in most cases. Specifically, when incorporating more BN layers in the adaptation, we could achiever better transfer results.
4.4 PRACTICAL APPLICATION FOR CLOUD DETECTION IN REMOTE SENSING IMAGES
In this section, we further demonstrate the effectiveness of AdaBN on a practical problem: Cloud Detection in Remote Sensing Images. Since remote sensing images are taken by different satellites with different sensors and resolutions, the captured images are visually different in texture, color, and value range distributions, as shown in Fig. 5. How to adapt a model trained on one satellite to another satellite images is naturally a domain adaptation problem.
Our task here is to identify cloud from the remote sensing images, which can be regarded as a semantic segmentation task. The experiment is taken under a self-collected dataset, which includes three image sets, from GF2, GF1 and Tianhui satellites. Each image set contains 635, 324 and 113 images with resolution over 6000x6000 pixels respectively. We name the three different datasets following the satellite names. GF2 dataset is used as the training dataset while GF1 and Tianhui datasets are for testing. We use a state-of-art semantic segmentation method (Chen et al., 2016a) as our baseline model.
The results on GF1 and Tianhui datasets are shown in Table 4. The relatively low results of the baseline method indicate that there exists large distribution disparity among images from different satellites. Thus, the significant improvement after applying AdaBN reveals the effectiveness of our method. Some of the visual results are shown in Fig. 6. Since other domain adaptation methods require either additional optimization steps and extra components (e.g. MMD) or post-processing distribution alignment (like CORAL), it is very hard to apply these methods from image classification to this large-size (6000x6000) segmentation problem. Comparatively, besides the effective performance, our method needs no extra parameters and very few computations over the whole adaptation process.
5 CONCLUSION AND FUTURE WORKS
In this paper, we have introduced a simple yet effective approach for domain adaptation on batch normalized neural networks. Besides its original uses, we have exploited another functionality of Batch Normalization (BN) layer: domain adaptation. The main idea is to replace the statistics of each BN layer in source domain with those in target domain. The proposed method is easy to implement and parameter-free, and it takes almost no effort to extend to multiple source domains and semi-supervised settings. Our method established new state-of-the-art results on both single and multiple source(s) domain adaptation settings on standard benchmarks. At last, the experiments on cloud detection for large-size remote sensing images further demonstrate the effectiveness of our method in practical use. We believe our method opens up a new direction for domain adaptation.
In contrary to other methods that use Maximum Mean Discrepancy (MMD) or domain confusion loss to update the weights in CNN for domain adaptation, our method only modifies the statistics of BN layer. Therefore, our method is fully complementary to other existing deep learning based methods. It is interesting to see how these different methods can be unified under one framework.

Overall I think this is an interesting paper which shows empirical performance improvement over baselines. However, my main concern with the paper is regarding its technical depth, as the gist of the paper can be summarized as the following: instead of keeping the batch norm mean and bias estimation over the whole model, estimate them on a per-domain basis. I am not sure if this is novel, as this is a natural extension of the original batch normalization paper. Overall I think this paper is more fit as a short workshop presentation rather than a full conference paper.
Detailed comments:
Section 3.1: I respectfully disagree that the core idea of BN is to align the distribution of training data. It does this as a side effect, but the major purpose of BN is to properly control the scale of the gradient so we can train very deep models without the problem of vanishing gradients. It is plausible that intermediate features from different datasets naturally show as different groups in a t-SNE embedding. This is not the particular feature of batch normalization: visualizing a set of intermediate features with AlexNet and one gets the same results. So the premise in section 3.1 is not accurate.
Section 3.3: I have the same concern as the other reviewer. It seems to be quite detatched from the general idea of AdaBN. Equation 2 presents an obvious argument that the combined BN-fully_connected layer forms a linear transform, which is true in the original BN case and in this case as well. I do not think it adds much theoretical depth to the paper. (In general the novelty of this paper seems low)
Experiments:
- section 4.3.1 is not an accurate measure of the "effectiveness" of the proposed method, but a verification of a simple fact: previously, we normalize the source domain features into a Gaussian distribution. the proposed method is explicitly normalizing the target domain features into the same Gaussian distribution as well. So, it is obvious that the KL divergence between these two distributions are closer - in fact, one is *explicitly* making them close. However, this does not directly correlate to the effectiveness of the final classification performance.
- section 4.3.2: the sensitivity analysis is a very interesting read, as it suggests that only a very few number of images are needed to account for the domain shift in the AdaBN parameter estimation. This seems to suggest that a single "whitening" operation is already good enough to offset the domain bias (in both cases shown, a single batch is sufficient to recover about 80% of the performance gain, although I cannot get data for even smaller number of examples from the figure). It would thus be useful to have a comparison between these approaches, and also a detailed analysis of the effect from each layer of the model - the current analysis seems a bit thin.


We focus on generative autoencoders, such as variational or adversarial autoencoders, which jointly learn a generative model alongside an inference model. Generative autoencoders are those which are trained to softly enforce a prior on the latent distribution learned by the inference model. We call the distribution to which the inference model maps observed samples, the learned latent distribution, which may not be consistent with the prior. We formulate a Markov chain Monte Carlo (MCMC) sampling process, equivalent to iteratively decoding and encoding, which allows us to sample from the learned latent distribution. Since, the generative model learns to map from the learned latent distribution, rather than the prior, we may use MCMC to improve the quality of samples drawn from the generative model, especially when the learned latent distribution is far from the prior. Using MCMC sampling, we are able to reveal previously unseen differences between generative autoencoders trained either with or without a denoising criterion.
1 INTRODUCTION
Unsupervised learning has benefited greatly from the introduction of deep generative models. In particular, the introduction of generative adversarial networks (GANs) (Goodfellow et al., 2014) and variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) has led to a plethora of research into learning latent variable models that are capable of generating data from complex distributions, including the space of natural images (Radford et al., 2015). Both of these models, and their extensions, operate by placing a prior distribution, P (Z), over a latent space Z ⊆ Rb, and learn mappings from the latent space, Z, to the space of the observed data, X ⊆ Ra. We are interested in autoencoding generative models, models which learn not just the generative mapping Z 7→ X , but also the inferential mapping X 7→ Z. Specifically, we define generative autoencoders as autoencoders which softly constrain their latent distribution, to match a specified prior distribution, P (Z). This is achieved by minimising a loss, Lprior, between the latent distribution and the prior. This includes VAEs (Kingma & Welling, 2014; Rezende et al., 2014), extensions of VAEs (Kingma et al., 2016), and also adversarial autoencoders (AAEs) (Makhzani et al., 2015). Whilst other autoencoders also learn an encoding function, e : Ra → Z, together with a decoding function, d : Rb → X , the latent space is not necessarily constrained to conform to a specified probability distribution. This is the key distinction for generative autoencoders; both e and d can still be deterministic functions (Makhzani et al., 2015).
The functions e and d are defined for any input from Ra and Rb respectively, however the outputs of the functions may be constrained practically by the type of functions that e and d are, such that e maps to Z ⊆ Rb and d maps to X ⊆ Ra. During training however, the encoder, e is only fed with training data samples, x ∈ X and the decoder, d is only fed with samples from the encoder, z ∈ Z, and so the encoder and decoder learn mappings between X and Z.
The process of encoding and decoding may be interpreted as sampling the conditional probabilities Qφ(Z|X) and Pθ(X|Z) respectively. The conditional distributions may be sampled using the encoding and decoding functions e(X;φ) and d(Z; θ), where φ and θ are learned parameters of the
encoding and decoding functions respectively. The decoder of a generative autoencoder may be used to generate new samples that are consistent with the data. There are two traditional approaches for sampling generative autoencoders:
Approach 1 (Bengio et al., 2014):
x0 ∼ P (X), z0 ∼ Qφ(Z|X = x0), x1 ∼ Pθ(X|Z = z0)
where P (X) is the data generating distribution. However, this approach is likely to generate samples similar to those in the training data, rather than generating novel samples that are consistent with the training data.
Approach 2 (Kingma & Welling, 2014; Makhzani et al., 2015; Rezende et al., 2014): z0 ∼ P (Z), x0 ∼ Pθ(X|Z = z0)
where P (Z) is the prior distribution enforced during training and Pθ(X|Z) is the decoder trained to map samples drawn from Qφ(Z|X) to samples consistent with P (X). This approach assumes that∫ Qφ(Z|X)P (X)dX = P (Z), suggesting that the encoder maps all data samples from P (X)
to a distribution that matches the prior distribution, P (Z). However, it is not always true that∫ Qφ(Z|X)P (X)dX = P (Z). Rather Qφ(Z|X) maps data samples to a distribution which we
call, P̂ (Z): ∫ Qφ(Z|X)P (X)dX = P̂ (Z)
where it is not necessarily true that P̂ (Z) = P (Z) because the prior is only softly enforced. The decoder, on the other hand, is trained to map encoded data samples (i.e. samples from∫ Qφ(Z|X)P (X)dX) to samples from X which have the distribution P (X). If the encoder maps
observed samples to latent samples with the distribution P̂ (Z), rather than the desired prior distribution, P (Z), then: ∫
Pθ(X|Z)P (Z)dZ 6= P (X)
This suggests that samples drawn from the decoder, Pθ(X|Z), conditioned on samples drawn from the prior, P (Z), may not be consistent with the data generating distribution, P (X). However, by conditioning on P̂ (Z): ∫
Pθ(X|Z)P̂ (Z)dZ = P (X)
This suggests that to obtain more realistic generations, latent samples should be drawn via z ∼ P̂ (Z) rather than z ∼ P (Z), followed by x ∼ Pθ(X|Z). A limited number of latent samples may be drawn from P̂ (Z) using the first two steps in Approach 1 - however this has the drawbacks discussed in Approach 1. We introduce an alternative method for sampling from P̂ (Z) which does not have the same drawbacks.
Our main contribution is the formulation of a Markov chain Monte Carlo (MCMC) sampling process for generative autoencoders, which allows us to sample from P̂ (Z). By iteratively sampling the chain, starting from an arbitrary zt=0 ∈ Rb, the chain converges to zt→∞ ∼ P̂ (Z), allowing us to draw latent samples from P̂ (Z) after several steps of MCMC sampling. From a practical perspective, this is achieved by iteratively decoding and encoding, which may be easily applied to existing generative autoencoders. Because P̂ (Z) is optimised to be close to P (Z), the initial sample, zt=0 can be drawn from P (Z), improving the quality of the samples within a few iterations.
When interpolating between latent encodings, there is no guarantee that z stays within high density regions of P̂ (Z). Previously, this has been addressed by using spherical, rather than linear interpolation of the high dimensional Z space (White, 2016). However, this approach attempts to keep z
within P (Z), rather than trying to sample from P̂ (Z). By instead applying several steps of MCMC sampling to the interpolated z samples before sampling Pθ(X|Z), unrealistic artifacts can be reduced (see Figure 2). Whilst most methods that aim to generate realistic samples from X rely on adjusting encodings of the observed data (White, 2016), our use of MCMC allows us to walk any latent sample to more probable regions of the learned latent distribution, resulting in more convincing generations. We demonstrate that the use of MCMC sampling improves generations from both VAEs and AAEs with high-dimensional Z; this is important as previous studies have shown that the dimensionality of Z should be scaled with the intrinsic latent dimensionality of the observed data.
Our second contribution is the modification of the proposed transition operator for the MCMC sampling process to denoising generative autoencoders. These are generative autoencoders trained using a denoising criterion, (Seung, 1997; Vincent et al., 2008). We reformulate our original MCMC sampling process to incorporate the noising and denoising processes, allowing us to use MCMC sampling on denoising generative autoencoders. We apply this sampling technique to two models. The first is the denoising VAE (DVAE) introduced by Im et al. (2015). We found that MCMC sampling revealed benefits of the denoising criterion. The second model is a denoising AAE (DAAE), constructed by applying the denoising criterion to the AAE. There were no modifications to the cost function. For both the DVAE and the DAAE, the effects of the denoising crtierion were not immediately obvious from the initial samples. Training generative autoencoders with a denoising criterion reduced visual artefacts found both in generations and in interpolations. The effect of the denoising criterion was revealed when sampling the denoising models using MCMC sampling.
2 BACKGROUND
One of the main tasks in machine learning is to learn explanatory factors for observed data, commonly known as inference. That is, given a data sample x ∈ X ⊆ Ra, we would like to find a corresponding latent encoding z ∈ Z ⊆ Rb. Another task is to learn the inverse, generative mapping from a given z to a corresponding x. In general, coming up with a suitable criterion for learning these mappings is difficult. Autoencoders solve both tasks efficiently by jointly learning an inferential mapping e(X;φ) and generative mapping d(Z; θ), using unlabelled data from X in a self-supervised fashion (Kingma & Welling, 2014). The basic objective of all autoencoders is to minimise a reconstruction cost, Lreconstruct, between the original data, X , and its reconstruction, d(e(X;φ); θ). Examples of Lreconstruct include the squared error loss, 12 ∑N n=1 ‖d(e(xn;φ); θ) − xn‖2, and the cross-entropy loss, H[P (X)‖P (d(e(X;φ); θ))] =
− ∑N n=1 xn log(d(e(xn;φ); θ)) + (1− xn) log(1− d(e(xn;φ); θ)).
Autoencoders may be cast into a probablistic framework, by considering samples x ∼ P (X) and z ∼ P (Z), and attempting to learn the conditional distributions Qφ(Z|X) and Pθ(X|Z) as e(X;φ) and d(Z; θ) respectively, with Lreconstruct representing the negative log-likelihood of the reconstruction given the encoding (Bengio, 2009). With any autoencoder, it is possible to create novel x ∈ X by passing a z ∈ Z through d(Z; θ), but we have no knowledge of appropriate choices of z beyond those obtained via e(X;φ). One solution is to constrain the latent space to which the encoding model maps observed samples. This can be achieved by an additional loss, Lprior, that penalises encodings far away from a specified prior distribution, P (Z). We now review two types of generative autoencoders, VAEs (Kingma & Welling, 2014; Rezende et al., 2014) and AAEs (Makhzani et al., 2015), which each take different approaches to formulating Lprior.
2.1 GENERATIVE AUTOENCODERS
Consider the case where e is constructed with stochastic neurons that can produce outputs from a specified probability distribution, and Lprior is used to constrain the distribution of outputs to P (Z). This leaves the problem of estimating the gradient of the autoencoder over the expectation EQφ(Z|X), which would typically be addressed with a Monte Carlo method. VAEs sidestep this by constructing latent samples using a deterministic function and a source of noise, moving the source of stochasticity to an input, and leaving the network itself deterministic for standard gradient calculations—a technique commonly known as the reparameterisation trick (Kingma & Welling, 2014). e(X;φ) then consists of a deterministic function, erep(X;φ), that outputs parameters for a probability distribution, plus a source of noise. In the case where P (Z) is a diagonal covariance Gaussian, erep(X;φ)
maps x to a vector of means, µ ∈ Rb, and a vector of standard deviations, σ ∈ Rb+, with the noise ∼ N (0, I). Put together, the encoder outputs samples z = µ+ σ, where is the Hadamard product. VAEs attempt to make these samples from the encoder match up with P (Z) by using the KL divergence between the parameters for a probability distribution outputted by erep(X;φ), and the parameters for the prior distribution, giving Lprior = DKL[Qφ(Z|X)‖P (Z)]. A multivariate Gaussian has an analytical KL divergence that can be further simplified when considering the unit Gaussian, resulting in Lprior = 12 ∑N n=1 µ
2 + σ2 − log(σ2)− 1. Another approach is to deterministically output the encodings z. Rather than minimising a metric between probability distributions using their parameters, we can turn this into a density ratio estimation problem where the goal is to learn a conditional distribution, Qφ(Z|X), such that the distribution of the encoded data samples, P̂ (Z) = ∫ Qφ(Z|X)P (X)dX , matches the prior distribution, P (Z). The GAN framework solves this density ratio estimation problem by transforming it into a class estimation problem using two networks (Goodfellow et al., 2014). The first network in GAN training is the discriminator network, Dψ , which is trained to maximise the log probability of samples from the “real” distribution, z ∼ P (Z), and minimise the log probability of samples from the “fake” distribution, z ∼ Qφ(Z|X). In our case e(X;φ) plays the role of the second network, the generator network, Gφ, which generates the “fake” samples.1 The two networks compete in a minimax game, where Gφ receives gradients from Dψ such that it learns to better fool Dψ . The training objective for both networks is given by Lprior = argminφ argmaxψ EP (Z)[log(Dψ(Z))]+ EP (X)[log(1−Dψ(Gφ(X)))] = argminφ argmaxψ EP (Z)[log(Dψ(Z))] + EQφ(Z|X)P (X) log[1− Dψ(Z)]. This formulation can create problems during training, so instead Gφ is trained to minimise − log(Dψ(Gφ(X))), which provides the same fixed point of the dynamics of Gφ and Dψ . The result of applying the GAN framework to the encoder of an autoencoder is the deterministic AAE (Makhzani et al., 2015).
2.2 DENOISING AUTOENCODERS
In a more general viewpoint, generative autoencoders fulfill the purpose of learning useful representations of the observed data. Another widely used class of autoencoders that achieve this are denoising autoencoders (DAEs), which are motivated by the idea that learned features should be robust to “partial destruction of the input” (Vincent et al., 2008). Not only does this require encoding the inputs, but capturing the statistical dependencies between the inputs so that corrupted data can be recovered (see Figure 3). DAEs are presented with a corrupted version of the input, x̃ ∈ X̃ , but must still reconstruct the original input, x ∈ X , where the noisy inputs are created through sampling x̃ ∼ C(X̃|X), a corruption process. The denoising criterion, Ldenoise, can be applied to any type of autoencoder by replacing the straightforward reconstruction criterion, Lreconstruct(X, d(e(X;φ); θ)), with the reconstruction criterion applied to noisy inputs: Lreconstruct(X, d(e(X̃;φ); θ)). The encoder is now used to model samples drawn from Qφ(Z|X̃). As such, we can construct denoising generative autoencoders by training autoencoders to minimise Ldenoise + Lprior. One might expect to see differences in samples drawn from denoising generative autoencoders and their non-denoising counterparts. However, Figures 4 and 6 show that this is not the case. Im et al.
1We adapt the variables to better fit the conventions used in the context of autoencoders.
(2015) address the case of DVAEs, claiming that the noise mapping requires adjusting the original VAE objective function. Our work is orthogonal to theirs, and others which adjust the training or model (Kingma et al., 2016), as we focus purely on sampling from generative autoencoders after training. We claim that the existing practice of drawing samples from generative autoencoders conditioned on z ∼ P (Z) is suboptimal, and the quality of samples can be improved by instead conditioning on z ∼ P̂ (Z) via MCMC sampling.
3 MARKOV SAMPLING
We now consider the case of sampling from generative autoencoders, where d(Z; θ) is used to draw samples from Pθ(X|Z). In Section 1, we showed that it was important, when sampling Pθ(X|Z), to condition on z’s drawn from P̂ (Z), rather than P (Z) as is often done in practice. However, we now show that for any initial z0 ∈ Z0 = Rb, Markov sampling can be used to produce a chain of samples zt, such that as t → ∞, produces samples zt that are from the distribution P̂ (Z), which may be used to draw meaningful samples from Pθ(X|Z), conditioned on z ∼ P̂ (Z). To speed up convergence we can initialise z0 from a distribution close to P̂ (Z), by drawing z0 ∼ P (Z).
3.1 MARKOV SAMPLING PROCESS
A generative autoencoder can be sampled by the following process:
z0 ∈ Z0 = Rb, xt+1 ∼ Pθ(X|Zt), zt+1 ∼ Qφ(Z|Xt+1)
This allows us to define a Markov chain with the transition operator
T (Zt+1|Zt) = ∫ Qφ(Zt+1|X)Pθ(X|Zt)dX (1)
for t ≥ 0. Drawing samples according to the transition operator T (Zt+1|Zt) produces a Markov chain. For the transition operator to be homogeneous, the parameters of the encoding and decoding functions are fixed during sampling.
3.2 CONVERGENCE PROPERTIES
We now show that the stationary distribution of sampling from the Markov chain is P̂ (Z).
Theorem 1. If T (Zt+1|Zt) defines an ergodic Markov chain, {Z1, Z2...Zt}, then the chain will converge to a stationary distribution, Π(Z), from any arbitrary initial distribution. The stationary distribution Π(Z) = P̂ (Z).
The proof of Theorem 1 can be found in (Rosenthal, 2001).
Lemma 1. T (Zt+1|Zt) defines an ergodic Markov chain.
Proof. For a Markov chain to be ergodic it must be both irreducible (it is possible to get from any state to any other state in a finite number of steps) and aperiodic (it is possible to get from any state to any other state without having to pass through a cycle). To satisfy these requirements, it is more than sufficient to show that T (Zt+1|Zt) > 0, since every z ∈ Z would be reachable from every other z ∈ Z. We show that Pθ(X|Z) > 0 and Qφ(Z|X) > 0, giving T (Zt+1|Zt) > 0, providing the proof of this in Section A of the supplementary material.
Lemma 2. The stationary distribution of the chain defined by T (Zt+1|Zt) is Π(Z) = P̂ (Z).
Proof. For the transition operator defined in Equation (1), the asymptotic distribution to which T (Zt+1|Zt) converges to is P̂ (Z), because P̂ (Z) is, by definition, the marginal of the joint distributionQφ(Z|X)P (X), over which theLprior used to learn the conditional distributionQφ(Z|X).
Using Lemmas 1 and 2 with Theorem 1, we can say that the Markov chain defined by the transition operator in Equation (1) will produce a Markov chain that converges to the stationary distribution Π(Z) = P̂ (Z).
3.3 EXTENSION TO DENOISING GENERATIVE AUTOENCODERS
A denoising generative autoencoder can be sampled by the following process:
z0 ∈ Z0 = Rb, xt+1 ∼ Pθ(X|Zt), x̃t+1 ∼ C(X̃|Xt+1), zt+1 ∼ Qφ(Z|X̃t+1).
This allows us to define a Markov chain with the transition operator
T (Zt+1|Zt) = ∫ Qφ(Zt+1|X̃)C(X̃|X)Pθ(X|Zt)dXdX̃ (2)
for t ≥ 0. The same arguments for the proof of convergence of Equation (1) can be applied to Equation (2).
3.4 RELATED WORK
Our work is inspired by that of Bengio et al. (2013); denoising autoencoders are cast into a probabilistic framework, where Pθ(X|X̃) is the denoising (decoder) distribution and C(X̃|X) is the corruption (encoding) distribution. X̃ represents the space of corrupted samples. Bengio et al. (2013) define a transition operator of a Markov chain – using these conditional distributions – whose stationary distribution is P (X) under the assumption that Pθ(X|X̃) perfectly denoises samples. The chain is initialised with samples from the training data, and used to generate a chain of samples from P (X). This work was generalised to include a corruption process that mapped data samples to latent variables (Bengio et al., 2014), to create a new type of network called Generative Stochastic Networks (GSNs). However in GSNs (Bengio et al., 2014) the latent space is not regularised with a prior.
Our work is similar to several approaches proposed by Bengio et al. (2013; 2014) and Rezende et al. (Rezende et al., 2014). Both Bengio et al. and Rezende et al. define a transition operator in terms of Xt and Xt−1. Bengio et al. generate samples with an initial X0 drawn from the observed data, while Rezende et al. reconstruct samples from an X0 which is a corrupted version of a data sample. In contrasts to Bengio et al. and Rezende et al., in this work we define the transition operator in terms of Zt+1 and Zt, initialise samples with a Z0 that is drawn from a prior distribution we can directly sample from, and then sample X1 conditioned on Z0. Although the initial samples may be poor, we are likely to generate a novel X1 on the first step of MCMC sampling, which would not be achieved using Bengio et al.’s or Rezende et al.’s approach. We are able draw initial Z0 from a prior because we constrain P̂ (Z) to be close to a prior distribution P (Z); in Bengio et al. a latent space is either not explicitly modeled (Bengio et al., 2013) or it is not constrained (Bengio et al., 2014).
Further, Rezende et al. (2014) explicitly assume that the distribution of latent samples drawn from Qφ(Z|X) matches the prior, P (Z). Instead, we assume that samples drawn from Qφ(Z|X) have a distribution P̂ (Z) that does not necessarily match the prior, P (Z). We propose an alternative method for sampling P̂ (Z) in order to improve the quality of generated image samples. Our motivation is also different to Rezende et al. (2014) since we use sampling to generate improved, novel data samples, while they use sampling to denoise corrupted samples.
3.5 EFFECT OF REGULARISATION METHOD
The choice of Lprior may effect how much improvement can be gained when using MCMC sampling, assuming that the optimisation process converges to a reasonable solution. We first consider the case of VAEs, which minimise DKL[Qφ(Z|X)‖P (Z)]. Minimising this KL divergence penalises the model P̂ (Z) if it contains samples that are outside the support of the true distribution P (Z), which might mean that P̂ (Z) captures only a part of P (Z). This means that when sampling
P (Z), we may draw from a region that is not captured by P̂ (Z). This suggests that MCMC sampling can improve samples from trained VAEs by walking them towards denser regions in P̂ (Z).
Generally speaking, using the reverse KL divergence during training, DKL[P (Z)‖Qφ(Z|X)], penalises the model Qφ(Z|X) if P (Z) produces samples that are outside of the support of P̂ (Z). By minimising this KL divergence, most samples in P (Z) will likely be in P̂ (Z) as well. AAEs, on the other hand are regularised using the JS entropy, given by 12DKL[P (Z)‖ 1 2 (P (Z) + Qφ(Z|X))] + 1 2DKL[Qφ(Z|X)‖ 1 2 (P (Z) + Qφ(Z|X))]. Minimising this cost function attempts to find a compromise between the aforementioned extremes. However, this still suggests that some samples from P (Z) may lie outside P̂ (Z), and so we expect AAEs to also benefit from MCMC sampling.
4 EXPERIMENTS
4.1 MODELS
We utilise the deep convolutional GAN (DCGAN) (Radford et al., 2015) as a basis for our autoencoder models. Although the recommendations from Radford et al. (2015) are for standard GAN architectures, we adopt them as sensible defaults for an autoencoder, with our encoder mimicking the DCGAN’s discriminator, and our decoder mimicking the generator. The encoder uses strided convolutions rather than max-pooling, and the decoder uses fractionally-strided convolutions rather than a fixed upsampling. Each convolutional layer is succeeded by spatial batch normalisation (Ioffe & Szegedy, 2015) and ReLU nonlinearities, except for the top of the decoder which utilises a sigmoid function to constrain the output values between 0 and 1. We minimise the cross-entropy between the original and reconstructed images. Although this results in blurry images in regions which are ambiguous, such as hair detail, we opt not to use extra loss functions that improve the visual quality of generations (Larsen et al., 2015; Dosovitskiy & Brox, 2016; Lamb et al., 2016) to avoid confounding our results.
Although the AAE is capable of approximating complex probabilistic posteriors (Makhzani et al., 2015), we construct ours to output a deterministic Qφ(Z|X). As such, the final layer of the encoder part of our AAEs is a convolutional layer that deterministically outputs a latent sample, z. The adversary is a fully-connected network with dropout and leaky ReLU nonlinearities. erep(X;φ) of our VAEs have an output of twice the size, which corresponds to the means, µ, and standard deviations, σ, of a diagonal covariance Gaussian distribution. For all models our prior, P (Z), is a 200D isotropic Gaussian with zero mean and unit variance: N (0, I).
4.2 DATASETS
Our primary dataset is the (aligned and cropped) CelebA dataset, which consists of 200,000 images of celebrities (Liu et al., 2015). The DCGAN (Radford et al., 2015) was the first generative neural network model to show convincing novel samples from this dataset, and it has been used ever since as a qualitative benchmark due to the amount and quality of samples. In Figures 7 and 8 of the supplementary material, we also include results on the SVHN dataset, which consists of 100,000 images of house numbers extracted from Google Street view images (Netzer et al., 2011).
4.3 TRAINING & EVALUATION
For all datasets we perform the same preprocessing: cropping the centre to create a square image, then resizing to 64 × 64px. We train our generative autoencoders for 20 epochs on the training split of the datasets, using Adam (Kingma & Ba, 2014) with α = 0.0002, β1 = 0.5 and β2 = 0.999. The denoising generative autoencoders use the additive Gaussian noise mapping C(X̃|X) = N (X, 0.25I). All of our experiments were run using the Torch library (Collobert et al., 2011).2
For evaluation, we generate novel samples from the decoder using z initially sampled from P (Z); we also show spherical interpolations (White, 2016) between four images of the testing split, as depicted in Figure 2. We then perform several steps of MCMC sampling on the novel samples and interpolations. During this process, we use the training mode of batch normalisation (Ioffe &
2Example code is available at https://github.com/Kaixhin/Autoencoders.
Szegedy, 2015), i.e., we normalise the inputs using minibatch rather than population statistics, as the normalisation can partially compensate for poor initial inputs (see Figure 4) that are far from the training distribution. We compare novel samples between all models below, and leave further interpolation results to Figures 5 and 6 of the supplementary material.
4.4 SAMPLES
5 CONCLUSION
Autoencoders consist of a decoder, d(Z; θ) and an encoder, e(X;φ) function, where φ and θ are learned parameters. Functions e(X;φ) and d(Z; θ) may be used to draw samples from the conditional distributions Pθ(X|Z) andQφ(Z|X) (Bengio et al., 2014; 2013; Rezende et al., 2014), where X refers to the space of observed samples and Z refers to the space of latent samples. The encoder distribution, Qφ(Z|X), maps data samples from the data generating distribution, P (X), to a latent distribution, P̂ (Z). The decoder distribution, Pθ(X|Z), maps samples from P̂ (Z) to P (X). We are concerned with generative autoencoders, which we define to be a family of autoencoders where regularisation is used during training to encourage P̂ (Z) to be close to a known prior P (Z). Commonly it is assumed that P̂ (Z) and P (Z) are similar, such that samples from P (Z) may be used to sample a decoder Pθ(X|Z); we do not make the assumption that P̂ (Z) and P (Z) are “sufficiently close” (Rezende et al., 2014). Instead, we derive an MCMC process, whose stationary distribution is P̂ (Z), allowing us to directly draw samples from P̂ (Z). By conditioning on samples from P̂ (Z), samples drawn from x ∼ Pθ(X|Z) are more consistent with the training data. In our experiments, we compare samples x ∼ Pθ(X|Z = z0), z0 ∼ P (Z) to x ∼ Pθ(X|Z = zi) for i = {1, 5, 10}, where zi’s are obtained through MCMC sampling, to show that MCMC sampling improves initially poor samples (see Figure 4). We also show that artifacts in x samples induced by interpolations across the latent space can also be corrected by MCMC sampling see (Figure 2). We further validate our work by showing that the denoising properties of denoising generative autoencoders are best revealed by the use of MCMC sampling.
Our MCMC sampling process is straightforward, and can be applied easily to existing generative autoencoders. This technique is orthogonal to the use of more powerful posteriors in AAEs (Makhzani et al., 2015) and VAEs (Kingma et al., 2016), and the combination of both could result in further improvements in generative modeling. Finally, our basic MCMC process opens the doors to apply a large existing body of research on sampling methods to generative autoencoders.
ACKNOWLEDGEMENTS
We would like to acknowledge the EPSRC for funding through a Doctoral Training studentship and the support of the EPSRC CDT in Neurotechnology.
B CELEBA
B.1 INTERPOLATIONS
C STREET VIEW HOUSE NUMBERS
C.1 SAMPLES
C.2 INTERPOLATIONS

The authors argues that the standard ancestral sampling from stochastic autoencoders (such as the Variational Autoencoder and the Adversarial
Autoencoder) imposes the overly-restrictive constraint that the encoder distribution must marginally match the latent variable prior. They propose, as an alternative, a Markov Chain Monte Carlo approach that avoids the need to specify a simple parametric form for the prior.
The paper is not clearly written. Most critically, the notation the authors use is either deeply flawed, or there are simple misunderstanding with respect to the manipulations of probability distributions. For example, the authors seem to suggest that both distributions Q(Z|X) and Q(X|Z) are parametrized. For this to be true the model must either be trivially simple, or an energy-based model. There is no indication that they are speaking of an energy-based model. Another example of possible confusion is the statement that the ratio of distributions Q(Z|X)/P(Z) = 1. I believe this is supposed to be a ratio of marginals: Q(Z)/P(X) = 1. Overall, it seems like there is a confusion of what Q and P represent. The standard notation used in VAEs is to use P to represent the decoder distribution and
Q to represent the encoder distribution. This seems not to be how the authors are using these terms. Nor does it seem like there is a single consistent interpretation.
The empirical results consist entirely of qualitative results (samples and reconstructions) from a single dataset (CelebA). The samples are also not at all up to the quality of the SOTA models. The interpolations shown in Figures 1 and 3 both seems to look like interpolation in pixel space for both the VAE model and the proposed DVAE.