Generative models are important for several practical applications, from low level image processing tasks, to model-based planning in robotics. More generally, the study of generative models is motivated by the long-standing endeavor to model uncertainty and to discover structure by leveraging unlabeled data. Unfortunately, the lack of an ultimate task of interest has hindered progress in the field, as there is no established way to compare models and, often times, evaluation is based on mere visual inspection of samples drawn from such models. In this work, we aim at addressing this problem by introducing a new benchmark evaluation suite, dubbed GenEval. GenEval hosts a large array of distributions capturing many important properties of real datasets, yet in a controlled setting, such as lower intrinsic dimensionality, multi-modality, compositionality, independence and causal structure. Any model can be easily plugged for evaluation, provided it can generate samples. Our extensive evaluation suggests that different models have different strenghts, and that GenEval is a great tool to gain insights about how models and metrics work. We offer GenEval to the community 1 and believe that this benchmark will facilitate comparison and development of new generative models.
Overall, this is a thorough attempt at a system for evaluating various generative models on synthetic problems vaguely representative of the kinds of problems claimed to be covered by GANs. I think the approach and the conclusions drawn are mostly reasonable, with one major caveat discussed shortly.
I also think it would help in a revision to add evaluations of more recent successors to RealNVP, such as MAF (NIPS 2017, https://arxiv.org/abs/1705.07057 ), Glow ( https://arxiv.org/abs/1807.03039 ), and (although of course this paper came out concurrently with your submission) the promising FFJORD ( https://openreview.net/forum?id=rJxgknCcK7 ). The scale of comparison of GAN variants is also much smaller than that of Lucic et al. or their followup, Kurach et al. ( https://arxiv.org/abs/1807.04720 ), which is not cited here (and should be).
But primarily, I think there are some serious concerns with your choice of metrics that make the results as they are difficult to interpret.
"Note that OT is not a distance in the standard mathematical sense, as for instance the 'distance' between two sets of points sampled from the same distribution is not zero." -- You've confused some notions here. The Wasserstein-1 distance, which is a scalar times the variant of OT you use here, absolutely is a proper distance metric between distributions: W(P, Q) is a metric. But when you compute the OT distance between *samples*, OT(S, T) with S ~ P and T ~ Q, you're equivalently computing the distance W(\hat{P}, \hat{Q}) between the empirical distributions of the samples, \hat{P} = 1/N \sum_i \delta_{S_i} and the similar \hat{Q}, which of course are not the same thing as the source distributions themselves. You can, though, view OT(S, T) as an *estimator* of W(P, Q); the distance between *distributions* is what we actually care about.
It is well-known that these empirical distributions of samples \hat{P} converge to the true distribution P (in the Wasserstein sense, W(P, \hat{P})) exponentially slowly in the dimension, which is what your example about high-dimensional distributions demonstrates. Incidentally, this is exactly the example used in Arora et al. (ICML 2017, https://arxiv.org/abs/1703.00573 ). This means that, viewed as an estimator of the true distance between distributions, the empirical-distribution OT estimator is strongly biased. Thus it becomes very difficult to tell what the true OT value is at any sample size, and moreover this amount of bias might differ for different distribution pairs even at the same sample size, so *comparing* OT estimates at a fixed sample size is a tricky business. For example, in your Figure 2, when the "oracle" score is significantly more than zero, you know that all of your estimates are very strongly biased. There is not, as far as I know, any strong reason to suspect that this amount of bias should be comparable for different distribution pairs, making any conclusions drawn from these numbers suspect.
Your scheme you call "Two-Sample Test," first, should have a more specific name. Two-sample testing is an extremely broad field, with instances including the classical Kolmogorov-Smirnov test and t tests, the popular-in-ML kernel MMD-based tests, and even Wasserstein-based tests (e.g. https://arxiv.org/abs/1509.02237 ). Previous applications of these tests in GANs and generative models include Bounliphone et al. (ICLR 2016, https://arxiv.org/abs/1511.04581 ), Lopez-Paz and Oquab (2016 - which you cite without a venue but which was at ICLR 2017), Sutherland et al. (ICLR 2017, https://arxiv.org/abs/1611.04488 ), Huang et al. (2018), and more, using a variety of schemes. Your name for this should include "nearest neighbor" or something along those lines to avoid confusion.
Also, you call this an "extension of the original formulation," but in the common case where n(x) is more often right than wrong, your v is exactly \hat t - 1 of Lopez-Paz and Oquab; see their (2). If it's usually wrong, then v = 1 - \hat t; only when the signs differ per class does it significantly differ from theirs, and in any case I don't see a real motivation to put the absolute values for each class separately rather than just taking |\hat t - 1/2|.
Moreover, it's kind of crazy to term your v statistic a two-sample *test* -- you have nothing in there about its sampling distribution, which is key to hypothesis testing to obtain e.g. a p-value. (Maybe the variance of v is very different between different distributions; this is likely the case. In any case the variance will probably become extremely large as the dimension increases.) Comparing this score is thus difficult, but in any case calling it a "test" is potentially very misleading. You could, though, estimate the variance as described by Lopez-Paz and Oquab to construct a test.
Also: you can imagine the statistic v(S, T) as an estimator of the distance between distributions given as
  D(P, Q) = |1/2 - \int ( 1 if p(x) > q(x), 0 o.w.) p(x) dx|
          + |1/2 - \int (-1 if p(x) > q(x), 0 o.w.) q(x) dx|.
But v(S, T) is, like for the OT distance, a biased estimator of this distance, whose bias will get worse with the dimension. Thus, like with the OT, it's hard to meaningfully compare v(S, T) as an attempt to compare *distributions* based on D, which is what we actually care about. Here the oracle score does not show strong bias: assuming a reasonable number of samples, when P = Q the v estimator is always going be approximately 0. But this doesn't mean that other estimators aren't strongly biased, and indeed this is exactly what your Appendix C shows. The strong change in performance for KDE is somewhat hard to interpret, but maybe has something to do with the connection between KDE and NN-based methods?
Your log-likelihood score is an unbiased and asymptotically normal estimate of the true distribution score (the cross-entropy), so it's easy to compare. But it accounts only for a very small portion of comparing distributions.
There is at least one score in common use for this kind of evaluation with easy-to-compare estimators: the squared MMD. It has an easy-to-compute unbiased and asymptotically normal estimator, so it's easy to get confidence intervals for the true value between distributions at any sample size, making comparing the numbers based on a reasonable number of samples easy. There's also a well-devolped theory for how to construct p-values for a test if you want those; Bounliphone et al. above even developed a relative test to compare the MMDs of two models accounting for the correlations due to using the same "target" set, though if you use separate target sets (because you can easily sample more points from your synthetic distribution) then it's simpler. The choice of kernel does matter, but I think the median-heuristic Gaussian kernel would be a very reasonable score to add to your repertoire, and for particular distributions you also might be able to pick a better kernel (e.g. based on the causal factors when those exist). See also Binkowski et al. (ICLR 2018, https://arxiv.org/abs/1801.01401 ) for a detailed discussion of these issues in comparison to the FID score.
Using a metric whose estimation can be understood, and whose estimators can be reliably compared, is I think vital to any evaluation process. This also prevents issues like when RealNVP outperforms the oracle, which should be impossible with any proper evaluation metric.
Minor points:
- Why is Pedregosa et al. (2011) cited for fitting multivariate Gaussians by maximum likelihood? This is something that doesn't need a citation, especially not to scikit-learn, which doesn't even (I don't think) contain an implementation of fitting Gaussians beyond (np.mean(X, axis=0), np.cov(X, rowvar=False)).
- Mode coverage and related scores: this is based on assigning sample points to their single most likely clusters? I'd imagine that sometimes a model will output points far from any cluster, in which case the cluster that happens to be closest might happen to be the most likely, but it's strange to really count that point as part of that cluster for these scores. Or similarly, a point might be relatively evenly spaced between two clusters, in which case the assignment could be fairly arbitrary, again making these scores a little strange.