Masked Autoregressive Flow for Density Estimation
Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling the random numbers of the next model in the stack, we obtain a type of normalizing flow suitable for density estimation, which we call Masked Autoregressive Flow. This type of flow is closely related to Inverse Autoregressive Flow and is a generalization of Real NVP. Masked Autoregressive Flow achieves state-of-the-art performance in a range of general-purpose density estimation tasks.
This paper is generally well written and I enjoy reading it. It introduces an expressive density model called masked autoregressive flow (MAF) that stacks multiple MADE layers to form a normalizing flow. Although it seems a bit incremental since the techniques involved have been studied in IAF and MADE, this paper does a good job elaborating on different types of generative modeling and providing guidelines for their use cases. It also makes a connection between MAF and IAF. Only a few comments/questions below:
* It'd be helpful to motivate a bit more on the advantage of density models. What are the applications or downstream tasks that make density models more suitable than their alternatives such as VAEs or GANs? For example, does the proposed density model admit efficient algorithms for marginalization or other inference tasks?
* When applying multiple transformations in the normalizing flow, is it implied that the conditionals need not be Gaussian as in eq. 2 because the density is computed according to the density of u? When u is transformed from some lower-level random vectors, it could very likely be non-Gaussian. Do I understand correctly?
* Does the base density of an MAF (line 225) refer to the density of u in eq. 3 or the density of p(x_i|x_{1:i-1}) in eq. 1? I assumed it's the former.
* When comparing the results from MAF (5), MAF (10) and MAF MoG (5) on the data sets POWER and MNIST, MAF MoG outperforms the other two. I have the impression that if using multiple normalizing flow layers where each layer has multiple non-linear MADE layers, MAF should be universal without MoG. What's the authors' opinion on this?