Wasserstein Variational Inference
This paper introduces Wasserstein variational inference, a new form of approximate Bayesian inference based on optimal transport theory. Wasserstein variational inference uses a new family of divergences that includes both f-divergences and the Wasserstein distance as special cases. The gradients of the Wasserstein variational loss are obtained by backpropagating through the Sinkhorn iterations. This technique results in a very stable likelihood-free training method that can be used with implicit distributions and probabilistic programs. Using the Wasserstein variational inference framework, we introduce several new forms of autoencoders and test their robustness and performance against existing variational autoencoding techniques.
MAIN IDEAS  The paper introduces a new family of divergence measures, which it calls the c-Wasserstein family of divergences. In application (in this paper) the distance between a generative joint density p(x, z) and its approximation q(x, z) is minimized, where (x, z) ~ training data.  The "transport cost" function of the c-Wasserstein divergence depends on distributions p and q, as well as its usual arguments. Theoretical properties are provided (all of this is novel work), and a Monte Carlo estimate shown. The MC estimate is biased for a finite sample size, and the paper shows how to remove the bias in one specific case when p = q.  The gradients of this loss are obtainable by back-propagating through the so-called "Sinkhorn iterations" from optimal transport theory.  RELATIONSHIP TO PAST WORK  The paper builds nicely on Operator Variational Inference (from NIPS last year or the year before), and Renyi divergence variational inference (also at NIPS).  Note to authors: the paper is very heavily biased toward citing work from one inner circle: 7/10 of the first cited papers are from the same group (!). Surely the world is larger?   STRENGTHS  - A beautiful new theoretical umbrella that ties f-divergences and Wasserstein divergences together.  - Very well written paper.  WEAKNESSES  - In time gone by, or "the days still influenced by legacy effects", as the paper puts it in its opening paragraph, divergence measures in approximate inference would always be used with respect to a true partition function, exact moments, and so on. One would compare against a ground truth MCMC estimate, do some thermodynamic integration, etc. Now, the evaluation is very fuzzy: the authors look at the mean squared deviation in the latent space over 30 re-runs of each method. (We assume by mean squared deviation that 30 latent samples were drawn for each datapoint, for each method, and the deviation from the mean of those samples reported?) The paper could state why these metrics are insightful? A method that encodes x into a delta function at zero, would have a zero latent error, and so why is "outperforms" (Line 210) such a big deal?  - The flexibility through C^{pq} introduces another layer of complexity that a modeller has to mitigate.  - Is a bias introduced by running the Sinkhorn iterations for a finite number of steps?  - The paper is very terse (especially section 5.1) and   Typos:  Line 7 in Algorithm 1, sum(u_\tau ... Line 139, should it not map to the expected value of p(x | z)? Line 181, equation, end,  D(p(z) || q(z)) Line 224, J_e_nsen-Shannon