Improved Variational Inference with Inverse Autoregressive Flow
The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.
This paper proposes "inverse autoregressive flow", a technique that transforms a simple distribution into a more flexible posterior distribution for variational inference. IAF can be viewed as a simplified form of the normalizing flow (Rezende and Mohamed ,2015), where autoregressive functions are used as nonlinear transformations to make the posterior approximation for variational inference more flexible. The transformation has simple Jacobian determinant so it’s computationally efficient. In this paper, three variants of the IAF have been introduced and experimented: linear IAF, nonlinear IAF through MADE and nonlinear IAF through recurrent neural networks. The model is validated on MNIST and CIFAR-10 database. The proposed VAE+IAF outperforms vanilla VAE with simple diagonal Gaussian distribution in terms of log-likelihood. The samples generated from the VAE trained with IAF prior look less blurry compared to previous work using VAE.
This paper proposes a general solution to the variational auto-encoders (VAE) that enables more complicated and structured prior distribution. The experimental comparisons with previous work are solid enough. The paper is in general clearly written but the writing and formulation still needs polishing.  -- Line 150 -- 151 "where at each step we use a differently parametrized auto-regressive model \mathcal{N}(\mu^t, \sigma^t)" My understanding is that only z^0 is generated by diagonal Gaussian distribution \mathcal(\mu^0, \sigma^0), while the remaining z^t is obtained by transformation step z^t = L(x) \cdot z^{t-1}. Can you clarify the pipeline for generating a sample from VAE+IAF?  -- Line 158 -- Line 159: The current notation is quite confusing. The z^t has been defined in Eq.11 but not used here. It would be more clear if the author provide the final form rather than using "y" here. Also, "y" is often used for label rather than latent variable.   -- How many IAF steps are used in the experiments? The only clue I find is the notation "z^l" (line 227), where "l" is the l-th block (1 = 1..L). Please  elaborate on this point.  -- How does the proposed IAF compared to normalization flow (Rezende and Mohamed, 2015)? The previous work (Figure 3 of that paper) clearly demonstrated that proposed technique produces better approximation to true posterior. Please comment on the performance of IAF.