We propose a non-adversarial feature matching-based approach to train generative models. Our approach, Generative Feature Matching Networks (GFMN), leverages pretrained neural networks such as autoencoders and ConvNet classifiers to perform feature extraction. We perform an extensive number of experiments with different challenging datasets, including ImageNet. Our experimental results demonstrate that, due to the expressiveness of the features from pretrained ImageNet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art results for challenging benchmarks such as CIFAR10 and STL10.
This paper proposes to learn implicit generative models by a feature matching objective which forces the generator to produce samples that match the means of the data distribution in some fixed feature space, focusing on image generation and feature spaces given by pre-trained image classifiers.
On the positive side, the paper is well-written and easy to follow, the experiments are clearly described, and the evaluation shows the method can achieve good results on a few datasets. The method is nice in that, unlike GANs and the stability issues that come with them, it minimizes a single loss and requires only a single module, the generator.
On the other hand, the general applicability of the method is unclear, the novelty is somewhat limited, and the evaluation is missing a few important baselines. In detail:
1) The proposed objective was used as a GAN auxiliary objective in [Salimans et al., 2016] and further explored in [Warde-Farley & Bengio, 2017]. The novel bit here is that the proposed objective doesn’t include the standard GAN term (so no need for an adversarially-optimized discriminator), and the feature extractor is a fixed pre-trained classifier or encoder from an auto-encoder (rather than a discriminator). 
2) The method only forces the generator’s sample distribution to match the first moment (the mean) of the data distribution. While the paper shows that this can result in a generator that produces reasonably good samples in practice, it seems like this may have happened due to a “lucky” artifact of the chosen pre-trained feature extractors. For example, a degenerate generator that produces a single image whose features exactly match the mean would be a global optimum under this objective, equally good as a generator that exactly matches the data distribution. Perhaps no such image exists for the chosen pre-trained classifiers, but it’s nonetheless concerning that the objective does nothing to prevent this type of behavior in the general case. (This is similar to the mode collapse problem that often occurs with GAN training in practice, but at least a GAN generator is required to exactly match the full data distribution to achieve the global optimum of that objective.)
3) It’s unclear why the proposed ADAM-based Moving Average (AMA) updates are appropriate for estimate the mean features of the data distribution. Namely, unlike EMA updates, it’s not clear that this is an unbiased estimator (I suspect it’s not); i.e. that the expectation of the resulting estimates is actually the true mean of the dataset features.  It’s therefore not clear whether the stated objective is actually what’s being optimized when these AMA updates are used.
4) Related to (3), an important baseline which is not discussed is the true fixed mean of the dataset distribution. In Sec. 2.4 (on AMA) it’s claimed that “one would need large mini-batches for generating a good estimate of the mean features...this can easily result in memory issues”, but this is not true: one could trivially compute the full exact dataset mean of these fixed features by accumulating a sum over the dataset (e.g., one image a time, with minibatch size 1) and then dividing the result by the number of images in the dataset. Without this baseline, I can’t rule out that the method only works due to its reliance on the stochasticity of the dataset mean estimates to avoid the behavior described in (2), or even the fact that the estimates are biased due to the use of ADAM as described in (3).
5) The best results in Table 3 rely on initializing G with the weights of a decoder pretrained for autoencoding. However, the performance of the decoder itself with no additional training from the GFMN objective is not reported, so it’s possible that most of the result relies on *autoencoder* training rather than feature matching to get a good generator. This explanation seems especially plausible due to the fact that the learning rate is set to a miniscule value (5*10^-6 for ADAM, 1-2 orders of magnitude smaller than typical values). Without the generator pretraining, the next best CIFAR result is an Inception Score of 7.67, lower than the unsupervised result from [Warde-Farley & Bengio, 2017] of 7.72.
6) It is misleading to call the results based on ImageNet-pretrained models “unconditional” -- there is plenty of overlap in the supervision provided by the labeled images of the much larger ImageNet to CIFAR and other datasets explored here. This is especially true given that the reported metrics (Inception Score and FID) are themselves based on ImageNet-pretrained classifiers. If the results were instead compared to prior work on conditional generation (e.g. ProGAN (Karras et al., 2017), which reports CIFAR IS of 8.56), there would be a clear gap between these results and the state of the art.
Overall, the current version of the paper needs additional experiments and clarifying discussion to address these issues.
=======================================
REVISION
Based on the authors' responses, I withdraw points 3-5 from my original review. Thanks to the authors for the additional experiments. On (3), I indeed misunderstood where the moving average was being applied; thanks for the correction. On (4), the additional experiment using the global mean features for real data convinces me that the method does not rely on the stochasticity of the estimates. (Though, given that the global mean works just as well, it seems like it would be more efficient and arguably cleaner to simply have that be the main method. But this isn't a major issue.) On (5), I misread the learning rate specified for "using the autoencoder features" as being the learning rate for autoencoder *pretraining*; thanks for the correction. The added results in Appendix 11 do show that the pretrained decoder on its own does not produce good samples.
My biggest remaining concerns are with points (2) and (6) from my original review.
On (2), I did realize that features from multiple layers are used, but this doesn't theoretically prevent the generator from achieving the global minimum of the objective by producing a single image whose features are the mean of the features in the dataset. That being said, the paper shows that this doesn't tend to happen in practice with existing classifiers, which is an interesting empirical contribution. (It would be nice to also see ablation studies on this point, showing the results of training against features from single layers across the network.)
On (6), I'm still unconvinced that making use of ImageNet classifiers isn't providing something like a conditional training signal, and that using such classifiers isn't a bit of an "unfair advantage" vs. other methods when the metrics themselves are based on an ImageNet classifier. I realize that ImageNet and CIFAR have different label sets, but most if not all of the CIFAR classes are nonetheless represented -- in a finer-grained way -- in ImageNet. If ImageNet and CIFAR were really completely unrelated, an ImageNet classifier could not be used as an evaluation metric for CIFAR generators. (And yes, I saw the CelebA results, but for this dataset there's no quantitative comparison with prior work, and qualitatively, if the results are as good as or better than the 3 year old DCGAN results, I can't tell.)
On the other hand, given that the approach relies on these classifiers, I don't have a good suggestion for how to control for this and make the comparison with prior work completely fair. Still, it would be nice to see acknowledgment and discussion of this caveat in a future revision of the paper.
Overall, given that most of my concerns have been addressed with additional experiments and clarification, and that the paper is well-written and has some interesting results from its relatively simple approach, I've raised my rating to above acceptance threshold.