Neurally-Guided Procedural Models: Amortized Inference for Procedural Graphics Programs using Neural Networks
Probabilistic inference algorithms such as Sequential Monte Carlo (SMC) provide powerful tools for constraining procedural models in computer graphics, but they require many samples to produce desirable results. In this paper, we show how to create procedural models which learn how to satisfy constraints. We augment procedural models with neural networks which control how the model makes random choices based on the output it has generated thus far. We call such models neurally-guided procedural models. As a pre-computation, we train these models to maximize the likelihood of example outputs generated via SMC. They are then used as efficient SMC importance samplers, generating high-quality results with very few samples. We evaluate our method on L-system-like models with imagebased constraints. Given a desired quality threshold, neurally-guided models can generate satisfactory results up to 10x faster than unguided models.
The authors consider the task of generating samples from constrained procedural models for computer graphics, similar to that considered in the stochastically ordered SMC paper by Ritchie et. al. (as cited by authors). They present an inference procedure based on sequential Monte Carlo, similar to recent approaches proposed by Gu et al. (as cited) and Paige & Wood (see below for ref), in which data-driven importance proposals are learned by gradient descent on an importance sampling estimate of the evidence upper bound (i.e. an expectation-propagation). The main contribution is the proposed neural net architecture for the data-driven proposals, which is specific to procedural graphics problems. This architecture that shows good performance in the experiments under consideration.  
I think this paper is clearly written and makes some reasonable contributions, but could do with an editing pass to frame the approach a bit better and relate it to recent contributions that similarly seek to amortize inference in sequential models by training neural net proposals.  While the authors are the first (to my knowledge) to train NN proposals for SMC in a procedural graphics / probabilistic programming setting, their approach is of course very closely related the one developed by Gu and colleagues [1], which the authors cite and the one proposed by Paige and Wood [2], which they do not. I am assuming this work was done more or less concurrently and independently, and I in principle don't see a problem for this paper from the point of view of novelty. That said, the paper in its current revision still reads a bit like a graphics paper that pitches using NN proposals for SMC as its core idea. This is unfortunate, in that it would have been nice to see the authors relate their work to that done by others.  When it comes to amortizing inference in sequential problems by training neural nets, the key question (in my opinion) is how the design of the neural net should relate to the structure of the generative model. References [1] and [2] both present different different takes on this problem. I think the problem / model class under consideration by the authors has three some interesting properties, which I think they could discuss more explicitly  1. A partially generated image can be compared against the target image at any time. This means that the likelihood at any resampling step is actually a pretty good heuristic for the likelihood of the completed image. This is very different the setting in, say, state space models, where you evaluate a model against a previously unseen data point at every resampling step.   2. As far as I can tell, the models under consideration don't have any global parameters. This greatly simplifies the problem. For example, if you want to do sequential inference in a Gaussian mixture model in which you explicitly sample the model parameters, then you would need to design a neural net that predicts the mean  and covariance for each cluster given a dataset, which is a highly problem-specific task.  3. Because the authors are considering images, they can extract features in a coarse-to-fine like manner by averaging over pixels. This seems like a great thing to be able to do.   4. Another interesting thing about this setting (which the authors do discuss explicitly) is that the evaluation cost of the neural nets is not negligible. One therefore has to consider not only the amortized cost of training, but also the cost of prediction.   In terms of impact, I think the specificity of the problem class considered by the authors is both a good thing and a bad thing. On the one hand you might argue that there is little here that generalizes beyond procedural graphics tasks. On the other I think the paper serves as a nice case study.   On balance I am inclined to say this paper just about meets the threshold for acceptance.   Post Author Feedback edit: While I do agree with the authors that data driven proposals can, in principle, be of great use in vision as inverse graphics approaches, I am not immediately convinced that the particular neural net architecture proposed by the authors is directly applicable to this task.    Minor  > Our insight is that while such an oracle can be difficult or impossible to write by hand, it is possible to learn a program chain_neural that comes close.  - Let's get rid of "Our insight" and cite [1] and [2] here.  > When used for procedural modeling, however, it leads to models whose outputs lack diversity, making them unsuitable for generating visually-varied content.  - In this context it would be helpful to state SMC is an importance sampling method. In importance sampling methods the target density must be absolutely continuous with respect to the proposal density (see any introductory reference, the first that pops to mind is [3]), which in the authors language equates to the requirement that the proposal density is "less compact" than the target density.  - Why do the authors use a Gaussian to define the loss criterion in equation (3)? A priori, something like an exponential would seem more natural.   - In section 4.2, I would like the authors to define the notation ||∇ I_2(p)||=1, since a gradient is technically not defined for a binary image.   - The authors should call out the SI when positing equation 1.   - In the derivation of equation 1 in SI:          xs ∼ P_CM(x) -> xs ∼ P_CM(x | c)  - The authors should probably cite the Wake Sleep algorithm [4] as related work.    References  [1] Gu, S., Ghahramani, Z., and Turner, R. E. (2105) "Neural Adaptive Sequential Monte Carlo". Neural Information Processing Systems   [2] Paige, B., and Wood, F. (2016) Inference Networks for Sequential Monte Carlo in Graphical Models. International Conference for Machine Learning Research.   [3] Geyer, C. J. (2011). Importance Sampling, Simulated Tempering, and Umbrella Sampling. In Handbook of Markov Chain Monte Carlo Methods.  [4] Hinton, G. E., Dayan, P., Frey, B. J., and Neal, R. M. (1995). The "wake-sleep" algorithm for unsupervised neural networks. Science, 268(5214):1158–1161.