Learning to Pass Expectation Propagation Messages
Expectation Propagation (EP) is a popular approximate posterior inference algorithm that often provides a fast and accurate alternative to sampling-based methods. However, while the EP framework in theory allows for complex nonGaussian factors, there is still a significant practical barrier to using them within EP, because doing so requires the implementation of message update operators, which can be difficult and require hand-crafted approximations. In this work, we study the question of whether it is possible to automatically derive fast and accurate EP updates by learning a discriminative model (e.g., a neural network or random forest) to map EP message inputs to EP message outputs. We address the practical concerns that arise in the process, and we provide empirical analysis on several challenging and diverse factors, indicating that there is a space of factors where this approach appears promising.
This paper proposes to learning expectation 
      propagation (EP) message update operators from data that would enable fast 
      and efficient approximate inference in situations where computing these 
      operators is otherwise intractable. - The authors address an 
      interesting and relevant problem to the NIPS community and propose -- as 
      far as I know -- a novel method of addressing intractable message update 
      computations. The approach is reasonably straightforward: to use a more 
      computationally demanding approximation process (importance-sampling 
      driven Monte Carlo estimates) to generate data for use in learning a 
      function approximation (such as a neural network) to provide fast and 
      accurate approximate messages. - One of the things I appreciated 
      about this paper is its tone. The authors take pains to point out this 
      approach is not a panacea and describe the circumstances where such a 
      strategy would be effective. The experiments are designed to elucidate the 
      strengths and weaknesses of the proposed approach and not necessarily 
      restricted to trying to sell it. - Regarding the exposition of the 
      material: The paper was largely well written and easy to follow. However, 
      I do have one major issue with the organization of the paper. Please 
      refactor the paper and supplementary material to ensure that the paper 
      stands on its own. Eg. remove descriptions of plots that appear in the 
      supplementary material from the paper. This is essentially an abuse of the 
      page limit on NIPS papers. - Since computation time of the message 
      updates is an important motivation of the proposed scheme, a careful and 
      quantitative evaluation of it should be in the paper. It should include 
      details of training times as well as "test" times. - Another paper 
      relevant to the idea of using function approximation to increase the speed 
      of inference is: Efficient learning of deep boltzmann machines. R 
      Salakhutdinov, H Larochelle AISTATS 2010. One thing that the above 
      paper as well as refs [2], [3] and [11] have in common is their 
      consideration of the approximation of inference in the context of 
      parameter learning. Unless I missed something, the authors are rather 
      silent on the issue of how to use their proposed method in the context of 
      also learning parameters. Is this method practical in the inner-loop of 
      parameter learning? Do the authors envision a kind of warm restart where 
      relatively few training cycles would be required between learning steps? 
      It seems like the use of computationally intensive methods such as 
      importance sampling in the creation of the training data might render this 
      method impractical as an inner-loop to parameter learning. - The 
      authors mention that one way their approach differs from these methods 
      (mentioned above) is that they are exploiting modularity and that the 
      learned message update operators could be re-used in other contexts. This 
      is a good point that is likely mostly true, but isn't there some concern 
      that the distribution over incoming messages could change from context to 
      context and that the learned operator might not be optimal for some of 
      these contexts? - Figure 2 needs to be much better explained with 
      much more supporting text that it current has.