We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of L A T E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network and extrapolate drawings. Taken together these results are a step towards agents that induce useful, humanreadable programs from perceptual input.
Summary of paper:
This paper tackles the problem of inferring graphics programs from hand-drawn images by splitting it into two separate tasks:
(1) inferring trace sets (functions to use in the program) and
(2) program synthesis, using the results from (1).
The usefulness of this split is referred to as the trace hypothesis.
(1) is done by training a neural network on data [input = rendered image; output = trace sets] which is generated synthetically. During test time, a trace set is generated using a population-based method which samples and assigns weights to the guesses made by the neural network based on a similarity metric. Generalization to hand-drawn images is ensured by by learning the similarity metric.
(2) is done by feeding the trace set into a program synthesis tool of Solar Lezama. Since this is too slow, the authors design a search policy which proposes a restriction on the program search space, making it faster. The final loss for (2) in equation 3 takes into consideration the time taken to synthesize images in a search space. 
---
Quality: The experiments are thorough and it seems to work. The potential limitation is generalization to non-synthetic data.
Clarity: The high level idea is clear however some of the details are not clear.
Originality: This work is one of the first that tackles the problem described.
Significance: There are many ad-hoc choices made in the paper, making it hard to extract an underlying insight that makes things work. Is it the trace hypothesis? Or is it just that trying enough things made this work?
---
Some questions/comments:
- Regarding the trace set inference, the loss function during training and the subsequent use of SMC during test time is pretty unconventional. The use of the likelihood P_{\theta}[T | I] as a proposal, as the paper also acknowledges, is also unconventional. One way to look at this which could make it less unconventional is to pose the training phase as learning the proposal distribution in an amortized way (instead of maximizing likelihood) as, for example, in [1, 2].
- In section 2.1., the paper talks about learning the surrogate likelihood function L_{learned} in order to work well for actual hand drawings. This presumably stems from the problem of mismatch between the distribution of the synthetic data used for training and the actual hand drawings. But then L_{learned} is also learned from synthetic data. What makes this translate to non-synthetic data? Does this translate to non-synthetic data?
- What does "Intersection over Union" in Figure 8 mean?
- The details for 3.1 are not clear. In particular, what does t(\sigma | T) in equation 3 refer to? Time to synthesize all images in \sigma? Why is the concept of Bias-optimality important?
- It seems from Table 4 that by design, the learned policy for the program search space already limits the search space to programs with maximum depth of the abstract syntax tree of 3. What is the usual depth of an AST when using Sketch?
---
Minor Comments:
- In page 4, section 2.1: "But pixel-wise distance fares poorly... match the model's renders." and "Pixel-wise distance metrics are sensitive... search space over traces." seem to be saying the same thing
- End of page 5: \citep Polozov & Gulwani (2015)
- Page 6: \citep Solar Lezama (2008)
---
References
[1] Paige, B., & Wood, F. (2016). Inference Networks for Sequential Monte Carlo in Graphical Models. In Proceedings of the 33rd International Conference on Machine Learning, JMLR W&CP 48: 3040-3049.
[2] Le, T. A., Baydin, A. G., & Wood, F. (2017). Inference Compilation and Universal Probabilistic Programming. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (Vol. 54, pp. 1338–1348). Fort Lauderdale, FL, USA: PMLR.