LEARNING TO INFER AND EXECUTE 3D SHAPE PROGRAMS
Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higherlevel shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.
This paper presents an approach to infer shape programs given 3D models. The programs include placing and arranging predefined primitives in layouts and can be written as a program over a domain-specific language (DSL). 
The architecture consists of a recurrent network that encodes a 3D shape represented as a voxel grid and outputs the instructions using a LSTM decoder. The generation is two-step where the first step predicts a program ID and the second step predicts instructions within the program ID. This aspect wasn't completely clear to me, see questions below. A second module that renders the program to 3D is also implemented as a neural network in order to optimize the model parameter in a end-to-end manner by minimizing a reconstruction loss. 
The method is evaluated on 3D shape reconstruction tasks for chairs and tables categories of the ShapeNet dataset. The approach compares favorably to Tulsiani et al., which considers a shape to be composed of a fixed number of cuboids.
The paper is well written and investigates an important problem. But it is hard to tease of the contributions and the relative importance of various steps in the paper:
1. Structure search vs. prediction. How does the model perform relative to a search-based approach for program generation. That would be slower but perhaps more accurate. The prediction model can be thought of an amortized inference procedure for search problems. What advantages does the approach offer?
2. Choice of the DSL. Compared to CSG modeling instructions of Sharma et al. the proposed DSL is more targeted to the shape categories. While this restricts the space of programs (e.g., no intersection, subtraction operations are used) leading to better generation of chairs and tables, it also limits the range and generalization of the learned models to new categories. Some discussion and comparison with the choice of DSL would be useful. 
3. Is the neural render necessary -- Wouldn't it be easier to simply use automatic differentiation to compute gradients of the rendering engine? 
4. It is also not clear to me how having a differentiable renderer allows training in an end-to-end manner since the output space is discrete and variable length. In CSGNet (Sharma et al.) policy-gradient techniques were used to optimize the LSTM parameters. The details of the guided adaptation were unclear to me (Section 4.3).
5. Is the neural renderer reliable -- Is is not clear if the neural renderer can provide accurate gradients when the generated programs are incorrect since the model is trained on a clean samples. In practice this means that the encoder has to initialized well. Since the renderer is also learned, would it generalize to new programs within the same DSL but different distribution over primitives -- e.g., a set of tables that have many more legs. Some visualizations of the generated shapes from execution traces could be added, sampling programs from within and outside the program distributions used to train.
6. All the above points give an impression that the choice of DSL and careful initialization are important to get the model to work. Some discussion on how robust the model is to these choices would be useful. In other words how meaningful is the generalization from the supervised training set of templates chairs and tables? 
7. Missing baselines: The model is trained on 100,000 chairs and tables with full supervision. What is the performance of a nearest neighbor prediction algorithm? This is an important baseline that is missing. A comparison with a simplified CSGNet with shape primitives and union operations is also important. Tulsiani et al. consider unions but constrain that all instances have the same number of primitives which can lead to poor reconstruction results. Furthermore the training sets are likely different making evaluations unclear. I suggest training the following decoders on the same training set used in this approach (1) fixed set of cuboids (e.g., Tulsiani et al.), (2) A recurrent decoder with cuboids, (3) CSGNet (different primitives and operations), (4) a nearest neighbor predictor with the Hamming or Chamfer distance metric.