Stein Variational Gradient Descent as Moment Matching
Stein variational gradient descent (SVGD) is a non-parametric inference algorithm that evolves a set of particles to fit a given distribution of interest. We analyze the non-asymptotic properties of SVGD, showing that there exists a set of functions, which we call the Stein matching set, whose expectations are exactly estimated by any set of particles that satisfies the fixed point equation of SVGD. This set is the image of Stein operator applied on the feature maps of the positive definite kernel used in SVGD. Our results provide a theoretical framework for analyzing properties of SVGD with different kernels, shedding insight into optimal kernel choice. In particular, we show that SVGD with linear kernels yields exact estimation of means and variances on Gaussian distributions, while random Fourier features enable probabilistic bounds for distributional approximation. Our results offer a refreshing view of the classical inference problem as fitting Stein's identity or solving the Stein equation, which may motivate more efficient algorithms.
In "Stein Variational Gradient Descent as Moment Matching," the authors first introduce the algorithm known as Stein Variational Gradient Descent (SVGD). While some work has been done trying to provide a theoretical analysis of this method, the consistency of SVGD is largely still open for finite sizes of n. (In fact, the analysis is still completely open when the domain is not compact either.)  The authors of this paper make headway on this problem. By studying the fixed point solution to SVGD, they show there are a set of functions for which the the fixed point solution perfectly estimates their mean under the target distribution (they call this the Stein set of functions). They argue that using a polynomial kernel when the target is a Gaussian will force any fixed point solution of SVGD to exactly estimate the mean and covariance of the target distribution, assuming the SVGD solution points are full rank.  The major contribution of this paper is that by studying the properties of finite dimensional kernels, they are able to employ random Fourier features to provide a theoretical analysis of the fixed points for these "randomized" kernels. By using this formulation of a randomized kernel Stein discrepancy, the authors provide a VC-like bound on the (randomized) kernel Stein discrepancy's value at a fixed point solution that converges at a O(m^{-1/2}log^{3/2}m) rate, where m indexes the number of random Fourier features. Their results do also require the domain to be compact, but the work does appeal to the finite sample regime and thus is a solid theoretical finding.  My biggest critique of this paper are the assumptions in 3.4. Because they have modified the kernel with a randomized, low rank approximation, their results do not immediately carry over to the non-randomized version. For example, for the Gaussian kernel and Gaussian target on a truncated or unbounded domain, can the assumptions of 3.4 be checked? I would be very interested in knowing what conditions on n (the number of SVGD samples), m, the target distribution P, and the kernel k are enough to assure the first assumption of 3.4 holds. While I would love to see this included in the paper, I think the rest of the paper is strong enough to merit publication at the conference.  ** Quality  L30-31, 110-111: Perhaps it is worth emphasizing here that these results typically only hold on compact domains as well?  Eqn 2, L103, Eqn 9, etc.: Isn't k the kernel for H_0 and thus how can the Stein operator be defined for a R-valued function?  Eqn 8: Where is P_{x_j} defined? I think it might be more clear to leave P_x for this.  L158-171: These lines seem a bit too generic to be included in this paper in my opinion. There are formulations which are solutions to Stein's equation, e.g., see Theorem 5 of Gorham et al "Measuring Sample Quality with Diffusions." Studying solutions to Stein's equation is an active field of research.  L224-225: I can't tell how strong this assumption is. Most importantly, are fixed point solutions for SVGD in the original kernel  "close" to the fixed point solutions to SVGD in the random Fourier kernel? That piece seems necessary to push the theory forward for a non-randomized kernel.  ** Clarity  The paper is for the most part clearly written. There is some sloppiness where the authors commonly apply the Stein operator to a R-valued function, but otherwise the ideas are well explained. I recommend the authors address this issue throughout the paper.  ** Originality This paper does identify some new ways to view the kernel Stein discrepancy and uses those to provide a finite sample analysis of a related, randomized problem.  The theory here does only apply to these "randomized" kernels (via the random Fourier formulation), while the kernels used in practice are not random. Hence there is a little left to be shown, in particular, that the fixed point solutions are not too different between the randomized kernel and the original (nonrandom) kernel. See the last paragraph of the summary section.  ** Signifance SVGD has become a popular (though expensive) black box method for doing posterior inference, despite it having no great theoretical guarantees. This paper provides the best analysis of this method to date in the finite sample regime and thus is a nice contribution.