Neural Adaptive Sequential Monte Carlo
Sequential Monte Carlo (SMC), or particle filtering, is a popular class of methods for sampling from an intractable target distribution using a sequence of simpler intermediate distributions. Like other importance sampling-based methods, performance is critically dependent on the proposal distribution: a bad proposal can lead to arbitrarily inaccurate estimates of the target distribution. This paper presents a new method for automatically adapting the proposal using an approximation of the Kullback-Leibler divergence between the true posterior and the proposal distribution. The method is very flexible, applicable to any parameterized proposal distribution and it supports online and batch variants. We use the new framework to adapt powerful proposal distributions with rich parameterizations based upon neural networks leading to Neural Adaptive Sequential Monte Carlo (NASMC). Experiments indicate that NASMC significantly improves inference in a non-linear state space model outperforming adaptive proposal methods including the Extended Kalman and Unscented Particle Filters. Experiments also indicate that improved inference translates into improved parameter learning when NASMC is used as a subroutine of Particle Marginal Metropolis Hastings. Finally we show that NASMC is able to train a latent variable recurrent neural network (LV-RNN) achieving results that compete with the state-of-the-art for polymorphic music modelling. NASMC can be seen as bridging the gap between adaptive SMC methods and the recent work in scalable, black-box variational inference.
The paper proposes to learn proposal distributions for sequential Monte Carlo (SMC) in nonlinear state-space models (SSMs). The method (NASMC) parameterizes the proposals as (recurrent) neural networks (RNNs) which are trained to fit the filtering distribution obtained with SMC. The approach is evaluated in three different setups: (a) state inference; (b)Bayesian learning (inference) of model parameters; (c) ML parameter learning with approximate (SMC) inference as inner loop. Results suggest that the the adaptive proposals outperform naive proposals (the prior) and also techniques such as the extended Kalman Particle filter and the Unscented Particle Filter (where the latter two are applicable). For ML learning of NN SSMs the approach performs similar to recently proposed stochastic variational approaches.The paper is well motivated and clearly written. In addressing the question of efficient, scalable inference in latent variable SSMs the paper considers an important and timely question. The proposed solution is not extremely original given that there has recently been a large body of work on "learning inference". Even though much of this work has focused on variational approximations the specific idea of learning a NN proposal for importance sampling -- in the non-sequential case --has been explored e.g. in ref [26], and I think the approach is a particular instance of the cross entropy method but with NNs (e.g. Rubinstein & Kroese, 2004; [9]). Nevertheless, to my knowledge the particular setup in the paper has not been explored before, and more generally, I am not aware of much of the recent "learning inference" work being applied to time series models. Also, the proposed approach seems conceptually simple and general, and therefore of practical value.As far as I can judge the authors have put a good effort into evaluating the proposed scheme in the context of different inference problems, covering different use-cases of SMC (including relatively low-dimensional benchmark systems as well as training a stochastic RNN on music data). On the low-dimensional benchmark data the results suggest that the learned proposal distributions can indeed be advantageous, at least compared to simple proposals such as the prior. The results for learning stochastic RNNs are less clear cut, with NASMC performing roughly on par with models of the same class trained using a variational approximation to the posterior. Two questions the evaluation leaves at least partly open is whether we can actually expect the method to scale to large state spaces and whether it provides any advantages over variational inference networks in practice. I think the experimental section could be strengthened in two ways: Firstly, by investigating how, for a known models, the quality of the learned proposal scales with the complexity of the posterior, in particular with the dimensionality of the latent space. Secondly, it I think it would be useful to include a smaller learning experiment, on which you test different proposals, and also variational inference. In the stochastic RNN learning experiments in 5.4 it seems that neither approach to train models with stochastic latent variables does especially well (compared to RNN-NADE). But the insights that we can gain from this are in my eyes somewhat limited as they confound potential issues due to the choice of model class with potential failures of inference. (A simpler learning task may also reveal the bias of VB in learning that you discuss in section 6.) In general, I think it would be useful to study and discuss limitations and failure cases of the approach in more detail. For instance, one worry that one might have about the approach is the fact that, at least as explored in the paper, the training of the proposal relies on the samples from the posterior which is expensive to sample from exactly. Intuitively, this seems to lead to a chicken-and-egg problem: if the posterior is poorly sampled from in the first place, then a poor proposal will be learned - leading to even worse exploration of the posterior. To reduce variance (and allow online learning) the authors propose to use a samples from a filtering approximation to the true posterior as the target for the learned proposal. Again, it seems to me that this could have undesired and, importantly, difficult-to-detect consequences. Maybe these concerns are unfounded but I think it would be worth addressing them directly.  Some additional comments:* line 192: in what sense is the approach minimizing a global cost?* line 219: h_t is only defined in the appendix* Experimental evaluation: when you have a multimodal posterior does it make sense to look at the difference between the posterior *mean* and the ground truth? (This question applies to both the MD parameterization as well as the unimodal proposal...)* MD networks in section 5.1: did you check whether these networks actually learned multimodal proposals or are the mixtures used e.g. to represent heavy-tailed proposals (which could be parameterized more directly)? Furthermore, you argue that the true posterior is multimodal. It would be very instructive to see whether this multimodality is actually properly explored by NASMC.* how exactly did you use prior knowledge about the dynamics in the parameterization of you proposal networks?* section 5.1 the prior as proposal seems to do really poorly but using the prior dynamics in the context of a learned NN appears to give a big boost - how does that fit together? Maybe having a proposal that corresponds to the prior mean plus some heavy-tailed noise would already give you most of the gain? (I think it would be useful to include such a simple baseline.) Also, looking at the table in the appendix the evidence in favor of RNNs over NNs seems not so clear-cut. The main boost seems to come from incorporating prior dynamics, for either network type. But the difference between RNN-MD(-f) and NN-MD(-f) is actually quite small? [I'm finding the table in the main text almost a bit misleading.] Can you provide your "ground truth" LML estimate? * section 5.2: Why is \delta \theta inferred so poorly when the prior is used as proposal (compared to x)? Why is \theta not part of the latent state z_t (see appendix)? Maybe say why you can't apply EKPF / UPF? (Is this because they rely on gradients?) What does inference for sigma_v look like? I'd suggest including other proposal architectures in this experiment as well.* especially in Fig. 3, but also more generally, I think that a fair comparison of NASMC and a naive proposal such as the prior needs to take into account the computational cost of learning the proposal. Using 10 particles from the prior may perform poorly, but when taking into account the cost of learning the proposal you may be able to afford relatively more samples from the prior...* section 5.4 I assume that the generative model structure is exactly the same as for the model trained with SGVB? Otherwise, it would be hard to compare the results.* It would be very useful to know how sensitive the approach is to the right choice of hyperparameters (learning rates, # of particles for training the proposal, relative training of proposal vs. model network, etc.). Especially in high dimensions is there a danger of the proposal overfitting early in learning?* I think that one nice feature of SMC inference compared to SGVB (the way it is is typically implemented, i.e. with stochastic backprop) is that it can be used with discrete stochastic latent variables. Maybe it's worth pointing this out? * line 412ff, the comment regarding entropy: you can typically use an MC approximation when the entropy is not analytically tractable* Another reference you may want to include: Stuhlmueller et al (2013): Learning Stochastic Inverses