Learning to Infer Graphics Programs from Hand-Drawn Images
We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of L A T E X. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are a specification (spec) of what the graphics program needs to draw. We learn a model that uses program synthesis techniques to recover a graphics program from that spec. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network and extrapolate drawings.
This paper presents a model for synthesizing Latex code from (hand) drawn diagrams. It does so by combining two approaches: i) a NN-based model for generating graphics primitives that reconstruct the image, and ii) synthesize a program from these generated primitives to achieve a better/compressed/more general representation than just a ‘list’ of graphics primitives.  Strengths: - I quite enjoyed reading the paper as the work in it seems original, solid and complete (even a bit more than enough). The conclusions are well supported by concrete (and importantly, relevant) evaluation.  Weaknesses: - my only objection to the paper is that it packs up quite a lot of information, and because of the page-limits it doesn’t include all the details necessary to reconstruct the model. This means cuts were made, some of which are not warranted. Sure, the appendix is there, but the reader needs to get all the necessary details in the main body of the paper.  I quite enjoyed the paper, I think it’s definitely NIPS material, but it needs some additional polishing. I added my list of suggestions I think would help improve readability of the paper at the end of the review.  Questions: - I might have missed the point of section 4.2 - I see it as a (spacewise-costly) way to say “programs (as opposed to specs) are a better choice as they enable generalization/extrapolation via changing variable values“? What is the experiment there? If it’s just to show that by changing variables, one can extrapolate to different images, I would save space on 1/2 of Figure 9 and focus on lacking parts of the paper (190 - extrapolations produced by our system - how did the system produce those extrapolations? was there a human that changed variable values or is there something in the system enabling this?) - What is + in Figure 3? If elementwise addition, please specify that - Figure 4 caption explains why the number N of particles is not the same across models. However, that still doesn’t stop me from wondering whether there is a significant difference in performance in case all models are using the same number of particles. Do you have that information? - Line 54 mentions that the network can ‘derender’ images with beam search. Is beam search used or not? What is the size of the beam? Is beam search used for each of the N particles? - From what I understood, the model does not have access to previously generated commands. Can you confirm that? - The order of (generated) specs is irrelevant for rendering, but it is for the generation process. How do you cope with that? Do you use a particular order when training the model or do you permute the specs? - Table 2 - “;” denotes OR, right? I would personally use the BNF notation here and use “|” - 153 - ‘minimized 3 using gradient descent’  - how did you treat the fact that min is not differentiable? - Table 5 - this is evaluated on which problems exactly? The same 100 on which the policy was trained? - Please provide some DeepCoder -style baseline details - the same MLP structure? Applied to which engine? A search algorithm or Sketch? - I find 152 - 153 unclear - how did you synthesize minimum cost programs for each \sigma ? \sigma represents a space of possible solutions, no? - Please provide more details on how you trained L_learned - what is the dataset you trained it on (randomly selected pairs of images, sampled from the same pool of randomly generated images, with a twist)? How did you evaluate its performance? What is the error of that model? Was it treated as a regression or as a classification task? - Figure 7 introduces IoU. Is that the same IoU used in segmentation? If so, how does that apply here? Do you count the union/intersection of pixels? Please provide a citation where a reader can quickly understand that measure.   Suggestions: - full Table 3 is pretty, but it could easily be halved to save space for more important (missing!) details of the paper - the appendix is very bulky and not well structured. If you want to refer to the appendix, I would strongly suggest to refer to sections/subsections, otherwise a reader can easily get lost in finding the details - Section 2.1 starts strong, promising generalization to real hand drawings, but in the first sentence the reader realizes the model is trained on artificial data. Only in line 91 it says that the system is tested on hand-written figures. I would emphasize that from the beginning. - Line 108 - penalize using many different numerical constants - please provide a few examples before pointing to the supplement. - Line 154 - a bit more detail of the bilinear model would be necessary (how low-capacity?) - 175-176 - see supplement for details. You need to provide some details in the body of the paper! I want to get the idea how you model the prior from the paper and not the supplement - Table 5 - there’s a figure in the appendix which seems much more informative than this Table, consider using that one instead   The related work is well written, I would just suggest adding pix2code (https://arxiv.org/abs/1705.07962) and SPIRAL (https://arxiv.org/abs/1804.01118) for completeness.  UPDATE: I've read the author feedback and the other reviews. We all agree that the paper is dense, but we seem to like it nevertheless. This paper should be accepted, even as is because it's a valuable contribution, but I really hope authors will invest additional effort into clarifying the parts we found lacking.