LEARNING TO DESCRIBE SCENES WITH PROGRAMS
Human scene perception goes beyond recognizing a collection of objects and their pairwise relations. We understand higher-level, abstract regularities within the scene such as symmetry and repetition. Current vision recognition modules and scene representations fall short in this dimension. In this paper, we present scene programs, representing a scene via a symbolic program for its objects, attributes, and their relations. We also propose a model that infers such scene programs by exploiting a hierarchical, object-based scene representation. Experiments demonstrate that our model works well on synthetic data and transfers to real images with such compositional structure. The use of scene programs has enabled a number of applications, such as complex visual analogy-making and scene extrapolation.
[Overview]
In this paper, the authors proposed a new format of representation called scene programs, to describe the visual scenes. To extract the scene programs from scenes, the authors exploited the off-the-shelf object detection and segmentations model, mask r-cnn to extract all objects and the corresponding attributes from the images, and then detect groups for those objects, which are then used to generate the programs which matches the input scenes. The experiments are performed on a synthetics datasets which consists of multiple shapes with different attributes. The experiments shows that the proposed model can infer more accurate programs from the scenes, and those generated programs can be used to recover the input scenes more accurately. Besides, the authors also showed that the generated scene programs can be used for image editing and making visual analogy.
[Strengthes]
1. The authors proposed a new representation, called scene programs, to describe the visual scenes with some textual program. This is a new scene representation, which could be potentially used in various scenarios, such as the image synthesis in graphics.
2. The authors proposed a hierarchical method to model the structures in scenes. Specifically, the objects in a scene are first extracted and then grouped into multiple clusters, which will be used to guide the scene program synthesis. 
3. The experimental results demonstrate the effectiveness of the proposed method both qualitatively and quantitatively. The authors also showed the the programs generated  can be sued for image editing and cross-modality matching. 
[Weaknesses]
1. It is a bit unfair to compare the proposed method with the two baseline methods listed in Table 2. The authors used a pre-trained mask-rcnn to detect all objects and predict the attributes for all objects. However, the counterpart methods have no access to this supervision. Even in this case, CNN-LSTM seems achieve comparable performance on the first three metrics. 
2. The advantage of scene program compared with scene graph (Johnson et al) are not clear to me. Scene graph is also a symbolic representation for images. Also, for all the tasks mentioned in this paper, such as image editing and visual analogy, scene graph can probably also complete well. The authors should comment about the specific advantages of scene program in comparison with scene graph.
3. All the images shown in the paper seems arranged uniformly, which I think contains some bias to the proposed grouping strategy. I would like to see more diverse configurations of the foreground objects. It would be good to see if the proposed model can describe more complicated scenes.
[Summary]
This paper proposed a novel scene representations, called scene program. To extract the scene program, the authors proposed a hieratchical inference method. The resulting scene programs based on the proposed model outperforms several baseline models quantitatively. The authors also showed the proposed scene program is suitable for image editing and visual analogy making. However, as pointed above, there are some unclear points to me, especially the advantages of scene program compared with scene graph, and the representation power of scene program for complicated scenes.