Probabilistic Neural Programmed Networks for Scene Generation
In this paper we address the text to scene image generation problem. Generative models that capture the variability in complicated scenes containing rich semantics is a grand goal of image generation. Complicated scene images contain varied visual elements, compositional visual concepts, and complicated relations between objects. Generative models, as an analysis-by-synthesis process, should encompass the following three core components: 1) the generation process that composes the scene; 2) what are the primitive visual elements and how are they composed; 3) the rendering of abstract concepts into their pixel-level realizations. We propose PNPNet, a variational auto-encoder framework that addresses these three challenges: it flexibly composes images with a dynamic network structure, learns a set of distribution transformers that can compose distributions based on semantics, and decodes samples from these distributions into realistic images.
I have read the other reviews and the authors' rebuttal. My opinion remains that this is a strong submission and should be accepted. I appreciate the authors addressed each of my concerns and questions in sufficient detail. -------------------- This paper proposes a new, modular neural architecture for generating images of scenes composed of attributed objects. The model takes as input a description of the scene to be generated, in the form of a tree-structured 'program' describing the objects, their attributes and their spatial relations. Each elementary concept (object, attribute) is encoded into a distribution over spatial and visual appearance attributes (mu, sigma). These distributions are then composed--following the program tree structure--using a set of operators for combining attributes, describing objects with attributes, and laying out objects in the space. The final result is a final distribution which can be sampled to produce a latent vector representation of the entire scene; this latent representation is then decoded into an image via standard image decoder/generator networks. The model is trained using variational inference. The paper applies this model to generating scenes of colored MNIST digits as well as CLEVR-G scenes, where it often outperforms strong text-to-image baselines under an object-detection-based performance metric.  This paper makes a good contribution: taking ideas from Neural Module Networks and related work, used for analyzing and answering questions about images, and applying them to task of generating the images themselves. The proposed architecture is reasonable (though I wish, space permitting, more justifications for design decisions were given). The evaluation is appropriate, and the proposed model clearly produces better samples than alternative approaches. I have some concerns (see below), but overall I am currently in favor of accepting this paper.  - - - - - - -   I found much of the model to be insufficiently described / under-specified. The level of detail at which it is described is almost sufficient for the main paper. But there are also quite a few details which ought to be provided somewhere (at least in the supplemental) in order for this work to be reproducible:  -- Figure 2: What is w_layout? I infer that this is probably an embedding for a spatial relation concept, but this is never defined (or even mentioned anywhere in the paper text)  -- Figure 2: I believe (according to Equation 3) that *both* inputs should feed into the "Convs" block, yes?  -- Equations 2 and 3: It's not clear to which distributions these operators apply: both z_a and z_s, or just z_a? The way Figure 2 is drawn, it looks like they apply to both. But the f convolution in Equation 3 only makes sense for z_a, so I suspect that this is the case. This could use clearing up, and Figure 2 could be re-drawn to be more precise about it.  -- Equation 4: How are bounding boxes / A sampled? Figure 2 shows the sampler as conditioned on z_s, but how, precisely? What's the functional form of the distribution being sampled from?  -- Transform: I also don't quite follow how the distribution is resized. The bilinear interpolation makes it sound as if the parameter tensors are resampled onto different WxH resolutions--is this correct?  -- Equation 5: The generated bounding boxes don't appear in this equation, nor does w_layout. How is the background canvas placement accomplished? Does this step perhaps use Transform as a subroutine? As it stands, Equation 5 only shows the application of the "Refiner CNN," which is one small part of the overall diagram given in Figure 2.  -- Line 172: "we first map the semantic information into programmatic operations": is this process manual or automatic?  -- Equation 6: You've written the proposal distribution as q(z); I assume you mean q(z | x)? The form of this encoder is not mentioned anywhere.  -- There are other architectural details would would belong in the supplement (e.g. the LSTM encoder in the conditional prior).  The detector-based performance measure is a nice idea. I'm wondering, though--do you detect the 'spatial relation' attributes such as 'top' and 'left?' It's not obvious how to do that, if that is something your evaluation takes into account. If not, that should also be mentioned.  ColorMNIST: It would be good to see some qualitative results to go along with Table 1.  Visualizing the learned concepts: If I understand correctly, your model should also permit you to visualize attribute concepts such as 'shiny?' Do I have that right, and if so, what do these look like?  Figure 5: In the zero-shot experiment, which concept combinations were omitted from the training data? Knowing this would help to assess the errors that the different methods are making in the presented images.  Generalization to complex scenes: If I understand correctly, this experiment the model's ability to train on and generate larger scenes. Can the model generalize to larger scenes than the ones it was trained on? E.g. if you train a model on scenes with at most 4 objects but test it on scenes with up to 8, how does it behave? I would hope to see some graceful degradation, given the modular nature of the model.  Two related works which should probably be cited: (Generating structured scenes from text) "Image Generation from Scene Graphs," Johnson et al. CVPR 2018 (Disentangled representations) "Deep Convolutional Inverse Graphics Network," Kulkarni et al. NIPS 2015