Annealing Between Distributions by Averaging Moments
Many powerful Monte Carlo techniques for estimating partition functions, such as annealed importance sampling (AIS), are based on sampling from a sequence of intermediate distributions which interpolate between a tractable initial distribution and the intractable target distribution. The near-universal practice is to use geometric averages of the initial and target distributions, but alternative paths can perform substantially better. We present a novel sequence of intermediate distributions for exponential families defined by averaging the moments of the initial and target distributions. We analyze the asymptotic performance of both the geometric and moment averages paths and derive an asymptotically optimal piecewise linear schedule. AIS with moment averaging performs well empirically at estimating partition functions of restricted Boltzmann machines (RBMs), which form the building blocks of many deep learning models.
This paper presents an alternative method for choosing 
      intermediate distributions when performing AIS on distributions in the 
      exponential family. Experimental results are promising. These results may 
      be of significant practical benefit, as they could improve researchers' 
      ability to make objective comparisons between complex probabilistic 
      models. Overall I liked it. I did think the theoretical motivation for the 
      proposed intermediate distributions was somewhat lacking. Equation 
      3 -- cool! "high curvature" --> not high curvature. Rather, 
      large information distance per theta distance. Section 5.1 -- why 
      is the larger variance of 1437.89 *better*? I would expect larger variance 
      to be worse. The experimental results were entirely for second 
      order models, while the technique is proposed for arbitrary exponential 
      family models. It would be nice to see experiments for other choices of 
      sufficient statistic. page 7 - "substantially", say something 
      quantitative. Some speculation follows: It's never really 
      explained why interpolating between the sufficient statistics should be 
      better than interpolating between the parameter values. Eqs. 4 and 
      5 strongly suggest that the optimal path in parameter space is the 
      geodesic (minimal distance) path between the two parameterizations, with 
      the Fisher information as a metric. If I've done my algebra 
      correctly, then $ds/d\theta = G$. (write down s = \int dx p(x) dE/dtheta, 
      and take the gradient) If the path is linear in $s$, as you propose, then 
      for small steps $\Delta s$, $\Delta \theta = d\theta / ds \Delta s = 
      G^{-1} \Delta s$. An infinitesimal step along your proposed trajectory in 
      terms of \theta looks like the step in s times the inverse Fisher 
      information matrix. This is the same as moving towards the final \theta 
      taking into account the information geometry (with metric = Fisher 
      information matrix). That is, these updates have a functional form which 
      is nearly identical to the natural gradient update. Interpreting 
      your algorithm in terms of a natural gradient trajectory also opens up 
      some possible approximations/extensions. eg, roughly in order of 
      decreasing promise: - Apply the same technique to non-exponential 
      family models, where the sufficient statistics are not defined, but the 
      Fisher information still is. - Use well motivated metrics other than 
      the Fisher information -- the matrix natural gradient is likely to best 
      bet here (Amari, S.-I. (1998). Natural Gradient Works Efficiently in 
      Learning. Neural Computation) - Use an approximation to G^-1, which 
      can be computed much more cheaply (Sohl-Dickstein, J. (2012). The Natural 
      Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its 
      Use. arXiv:1205.1828v1.) - Iterate over choices for the entire 
      trajectory, in order to find the global shortest trajectory (with metric 
      G), rather than just taking the local shortest steps -- 
      Finally, here's a plug for reproducible science. You can become 
      more awesome by putting source code for experiments in the supplemental 
      material! When source code is shared for computational experiments, it has 
      a major positive impact on the usefulness of the authors' hard work for 
      future researchers, on the credibility of the work, and on the number of 
      citations earned. And as a reviewer it makes me more positively disposed 
      towards a paper.