Estimating the $\chi^2$-divergence with Monte Carlo: which distribution to sample from? Notation: let the $\chi^2$-divergence between $p, q$ be defined as
$$\chi^2 (p||q) := \int \left ( \frac{p(x)}{q(x)} \right )^2 q(x)\mathrm{d}x  -1 =  \int \frac{p(x)}{q(x)}  p(x)\mathrm{d}x - 1.  $$
Suppose $q$ is a fully known prior distribution, and $p$ is a posterior distribution known up to a normalizing constant but its normalizing constant can be estimated with a reliable Monte Carlo estimator with finite variance. If it helps simplify the problem, assume exact iid samples can be drawn from both distributions.

*

*Which estimator would be less variable? A Monte Carlo estimator with samples from $q$, or samples from $p$?

*Suppose I have samples from both $p$ and $q$. Is there a method for leveraging both samples to produce a more stable estimator?


Given $x_1,\ldots,x_n\sim p(x)$ and $y_1,\ldots,y_m\sim q(x)$, the multiple mixture estimator of Owen and Zhou (2000) is
$$\hat\xi=\sum_{i=1}^n \left( \frac{p(x_i)^2}{q(x_i)}\right)\frac{1}{np(x_i)+mq(x_i)}+\sum_{i=1}^m \left( \frac{p(y_i)^2}{q(y_i)}\right)\frac{1}{np(y_i)+mq(y_i)} - 1$$
It is unbiased for $\chi^2(p||q)$ and by mixing both densities it brings stability to the importance weights.
