FFJORD: FREE-FORM CONTINUOUS DYNAMICS FOR SCALABLE REVERSIBLE GENERATIVE MODELS
Reversible generative models map points from a simple distribution to a complex distribution through an easily invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the logdensity. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, improving the state-ofthe-art among exact likelihood methods with efficient sampling.
Reversible generative models use cheaply invertible neural networks to transform samples from a fixed base distribution. Examples include NICE (Dinh et al., 2014), Real NVP (Dinh et al., 2017), and Glow (Kingma & Dhariwal, 2018). These models are easy to sample from, and can be trained by maximum likelihood using the change of variables formula. However, this requires placing awkward restrictions on their architectures, such as partitioning dimensions or using rank one weight matrices, in order to avoid an O(D 3 ) cost determinant computation.
Summary:
This paper discusses an advance in the framework of normalizing flows for generative modeling, named FFJORD. The authors consider normalizing flows in the form of ordinary differential equations, as also discussed in [1]. Their contributions are two-fold: (1) they use an unbiased estimator of the likelihood of the model by approximating the trace of the jacobian with Hutchinson’s trace estimator, (2) they have implemented the required ODE solvers on GPUs. 
The models are evaluated on a density estimation task on tabular data and two image datasets (MNIST and CIFAR10), as well as on variational inference for auto-encoders, where the datasets MNIST, Omniglot, Freyfaces and Caltech Silhouettes are considered. 
The authors argue that the trace estimator, in combination with reverse-mode automatic differentiation to compute vector-Jacobian products, leads to a computational cost of O(D), instead of O(D^2) for the exact trace of the jacobian. 
They compare this to the cost of computing a Jacobian determinant for finite flows, which is O(D^3) in general. They argue that in general all works on finite flows have adjusted their architectures for the flows to avoid the O(D^3) complexity, and that FFJORD has no such restriction.
However, I would like the authors to comment on the following train of thought: autoregressive models, such as MAF, as well as IAF (inverse of an autoregressive model) do not require O(D^3) to compute jacobian determinants as the jacobian is of triangular form. Note however, they are still universal approximators if sufficient flows are applied, as any distribution can be factorized in an autoregressive manner. With this in mind, I find the red cross for MAF under free-form Jacobian slightly misleading. Perhaps I misunderstood something, so please clarify. 
Another topic that I would like the authors to comment on is efficiency and practical use. One of the main points that the authors seem to emphasise, is that contrary to autoregressive models, which require D passes through the model to sample a datapoint of size D, FFJORD is a ‘single-pass’ model, requiring only one pass through the model. They therefore indicate that they can do efficient sampling. However, for FFJORD every forward pass requires a pass through an ODE solver, which as the authors also state, can be very slow. I could imagine that this is still faster than an autoregressive model, but I doubt this is actually of comparable speed to a forward pass of a finite flow such as glow or realNVP. 
On the other hand, autoregressive models do not require D passes during training, whereas, if I understand correctly, FFJORD relies on two passes through ODE solvers, one for computing the loss, and a second to compute the gradient of the loss with respect to model parameters. So autoregressive models should train considerably faster. The authors do comment on the fact that FFJORD is slower than other models, but they do not give a hint as to how much slower it is. This would be of importance for practical use, and for other people to consider using FFJORD in future work. 
For the density estimation task, FFJORD does not have the best performance compared other baselines, except for MNIST, for which the overall best model was not evaluated (MAF-DDSF). For variational inference FFJORD is stated to outperform all other flows, but the models are only evaluated on the negative evidence lower bound, and not on the negative log-likehood (NLL). I suspect the NLL to be absent from the paper as it requires more computation, and this takes a long time for FFJORD. Without an evaluation on NLL the improvement over other methods is questionable. Even if the improvement still holds for the NLL, the relative improvement might not weigh heavily enough against increased runtime. FFJORD does require less memory than its competitors.
The improved runtime by implementing the ODE solvers on GPU versus the runtime on a CPU would be useful, given that this is listed as one of the main contributions.
Besides these questions/comments, I do think the idea of using Hutchinsons trace estimator is a valid contribution, and the experimental validation of continuous normalizing flows is of interest to the research community. Therefore, in my opinion, the community will benefit from the information in this paper, and it should be accepted. However I do wish for the authors to address the above questions as it would give a clearer view of the practical use of the proposed model. 
 
See below for comments and questions:
Quality
The paper has a good setup, and is well structured. The scope and limitations section is very much appreciated. 
Clarity
The paper is clearly written overall. The only section I can comment on is the related work section, which is not the best part of the paper. The division in normalizing flows and partitioned transformations is a bit odd. Partitioned transformations surely are also normalizing flows. Furthermore IAF by Kingma et al. is put in the box of autoregressive models, whereas it is the inverse of an autoregressive model, such that it does not have the D-pass sample problem. For a reader who is not too familiar with normalizing flows literature, I think this section is a little confusing. Furthermore, there is no related work discussed on continuous time flows, such as (but not limited to) [2].
Originality
The originality of the paper is not stellar, but sufficient for acceptance. 
Significance
The community can benefit from the experimental analysis of continuous time flows, and the GPU implementation of the ODE solver. Therefore I think this work is significant. 
Detailed questions/comments:
1. In section 4.2, as an additional downside to MAF-DDSF, the authors argue that sampling cannot be performed analytically. Since FFJORD needs to numerically propagate the ODE, I do not think FFJORD can sample analytically either. Is this correct?
2. The authors argue that they have no restriction on the architecture of the function f, even if they have O(D) estimation of the trace of the jacobian. However, they also say they make use of the bottle-neck trick to reduce the variance that arises due to Hutchinson’s estimate of the trace. This seems like a limitation on the architecture to me. Can the authors comment?
3. In B.1 in the appendix, the street view house numbers dataset is mentioned, but no results appear in the main text, why not?
4. In the results section, it is not clear to me which numbers of the baselines for different datasets are taken from other papers, and which numbers are obtained by the authors of this paper. Please clarify.
5. In the conclusions, when discussing future work, the authors state that they are interested in reducing the number of function evaluations in the ODE solvers. In various disciplines many people have worked on this problem for a long time. Do the authors think major improvements are soon to be made?
6. In section 5.2 the dependence of the number of function evaluations (NFE) on the data dimension D is discussed. As a thought experiment they use the fact that going from an isotropic gaussian distribution (in any D), to an isotropic gaussian distribution has a corresponding differential equation of zero. This should convince the reader that NFE is independent of D. However, this seems to me to be such a singular example, that I gain no insight from it, and it is not very convincing. Do the authors agree that this particular example does not add much? If not, please explain. 
[1] Chen et al. Neural ordinary differential equations. NIPS 2018
[2] Chen et al. Continuous-time flows for deep generative models.
**** EDIT *****
I have read the response of the authors and appreciate their clarifications and the additional information on the runtimes. See my response below for the concern that remains about the absence of the estimate of the log likelihood for the VAE experiments. Besides this issue, the other comments/answers were satisfactory, and I think this paper is of interest to the research community, so I will stick with my score.