Probabilistic models are often trained by maximum likelihood, which corresponds to minimizing a specific form of f -divergence between the model and data distribution. We derive an upper bound that holds for all f -divergences, showing the intuitive result that the divergence between two joint distributions is at least as great as the divergence between their corresponding marginals. Additionally, the f -divergence is not formally defined when two distributions have different supports. We thus propose a noisy version of f -divergence which is well defined in such situations. We demonstrate how the bound and the new version of f -divergence can be readily used to train complex probabilistic generative models of data and that the fitted model can depend significantly on the particular divergence used.
The paper proposes a method for training generative models with general f-divergences between the model and empirical distribution (the VAE and GAN objectives are captured as specific instantiations). The trick that leads to computational tractability involves utilizing a latent variable and optimizing the f-divergence between joint distributions which is an upper bound to the (desired) f-divergence between the marginal distributions. Distribution support issues are handled by convolving the data space with a blurring function. Empirical results on image datasets demonstrate that the additional training flexibility results in qualitatively different learned models. Specifically, optimizing the reverse KL (and/or Jensen-Shannon divergence) leads to sharper images, perhaps at the loss of some variance in the data distribution.
I liked the simplicity of the core idea, and appreciated the exposition being generally easy to follow. The application of upper bounds to general f-divergences for training generative models is novel as far as I know. My two issues are with the practicality of the implementation and evaluation methodology, both potentially affecting the significance of the work. Regarding practicality, it appears the training details are specific to each experiment. This begs the question of how sensitive the results are to these settings. Regarding the methodology, it would have been nice to see the method of Nowozin et al. (2016) applied in all experiments since this is a direct competitor to the proposed method. Moreover, the subjective nature of the results in the real dataset experiments makes it difficult to judge what the increased flexibility in training really provides - although I do note that the authors make this same point in the paper. Finally, given that model training is the paper's focus, an explicit discussion of computational cost was missed.
Even with these issues, however, I believe the paper makes a contribution to the important problem of fitting expressive generative models. In my opinion, a more rigorous and thorough experimental exploration would increase the value, but the paper demonstrates that training with alternative f-divergences is feasible.