We present a deep generative model, named Monge-Ampère flow, which builds on continuous-time gradient flow arising from the Monge-Ampère equation in optimal transport theory. The generative map from the latent space to the data space follows a dynamical system, where a learnable potential function guides a compressible fluid to flow towards the target density distribution. Training of the model amounts to solving an optimal control problem. The Monge-Ampère flow has tractable likelihoods and supports efficient sampling and inference. One can easily impose symmetry constraints in the generative model by designing suitable scalar potential functions. We apply the approach to unsupervised density estimation of the MNIST dataset and variational calculation of the two-dimensional Ising model at the critical point. This approach brings insights and techniques from Monge-Ampère equation, optimal transport, and fluid dynamics into reversible flow-based generative models.
This paper proposes a novel parameter-efficient generative modeling approach that is based on the Monge-Ampere equation. In the proposal, a feed-forward neural network is trained as an ODE integrator which solves (2) and (3) for a fixed time interval $[0,T]$, so that the distribution $p(x,t)$ at time 0 is a simple base distribution such as a Gaussian, and that at time $T$ mimics the target distribution.
[pros]
- The proposal provides a parameter-efficient approach to generative modeling, via parameter sharing in the depth direction.
- I think that the idea itself is quite interesting and that it is worth pursuing this direction further.
[cons]
- The Applications section is not convincing enough to demonstrate usefulness of the proposal as an approach to generative modeling.
- How the gradient-based learning in the proposal behaves is not discussed in this paper.
[quality]
How the gradient-based learning in the proposal behaves is not discussed. I understand that the non-convex nature of the loss function poses problems already in the conventional back-propagation learning of a multilayer neural network. On the other hand, in the proposal, the loss function (e.g., (4)) is further indirectly parameterized via $\varphi$. It would be nice if the parameterization of the loss in terms of $\varphi$ is regular in some sence.
[clarity]
Description of this paper is basically clear. In the author-date citation style employed in this paper, both the author names and publication year are enclosed in parentheses, with exception being the author names incorporated in the text. This paper does not follow the above standard convention for citation and thus poses strong resistance to the reader. For example, in the first line of the Introduction section, "Goodfellow et al. (2016)" should read "(Goodfellow et al., 2016)".
[originality]
The idea of considering the Monge-Ampere equation in its linearized form to formulate generative modeling seems original.
[significance]
In the experiment described in Section 4.1, it is not clear at all from the description here whether the learned system is capable of successfully generating MNIST-like fake images, which would question the significance of the proposal as a framework for generative modeling. It is well known that the KL divergence $D(P\|Q)$ tends to put more penalty when $P$ is large and $Q$ is small than the opposite. One can then expect in this experiment that it tolerates the model, appearing as $Q$ in $D(P\|Q)$, to put weights on regions where the data are scarce, which might result in generation of low-quality fake images. It would be nice if the authors provide figures showing samples generated via mapping of Gaussian samples with the learned system.
Also, in the experiment described in Section 4.2, I do not see its significance. It is nice to observe in Figure 4 that the loss function approaches the true free energy as well as that the snapshots generated by the model seem more or less realistic. My main concern however is regarding what the potential utilities of the proposal are in elucidating statistical-physical properties of a system. For example, it would be nice if the proposal could estimate the phase-transition point more easily and/or more accurately compared with alternative conventional approaches, but there is no such comparison presented in this paper, making the significance of this paper obscure.
Minor points:
The reference entitled "A proposal on machine learning via dynamical systems" would be better cited not as "E (2017)" but rather as "Weinan (2017)".
Page 6, line 10: the likelihoods of these sample(s)
----Updated after author feedback----
Upon reading the author feedback, I have downgraded my rating from 7 to 6, because the author feedback is not satisfactory to me in some respects. In my initial review, my comment on the experiment on MNIST is not on correlation between the maximum likelihood estimation and visual quality of generated images, on which the author feedback was based, but regarding the well-known property of the KL divergence due to its asymmetry between the two arguments. Also, regarding the experiment on the Ising model, the proposal in this paper provides an approximate sampler, whereas for example the MCMC provides an exact sampler with exponential slowing down in mixing under multimodal distributions. In statistical physics, one is interested in studying physical properties of the system, such as phase transition, with samples obtained from a sampler. In this regard, important questions are how good the samples are and how efficiently they are generated. As for the quality, it would have been nice if results of evaluated free energy as a function of inverse temperature (that is K_ij in the case here) were provided. The author feedback was, on the other hand, mainly explanation of general variational approach, of which I am aware.
I still think that this paper contains interesting contributions, and accordingly have put my rating above the threshold.