LAYOUTGAN: GENERATING GRAPHIC LAYOUTS WITH WIREFRAME DISCRIMINATORS
Layout is important for graphic design and scene generation. We propose a novel Generative Adversarial Network, called LayoutGAN, that synthesizes layouts by modeling geometric relations of different types of 2D elements. The generator of LayoutGAN takes as input a set of randomly-placed 2D graphic elements and uses self-attention modules to refine their labels and geometric parameters jointly to produce a realistic layout. Accurate alignment is critical for good layouts. We thus propose a novel differentiable wireframe rendering layer that maps the generated layout to a wireframe image, upon which a CNN-based discriminator is used to optimize the layouts in image space. We validate the effectiveness of Layout-GAN in various experiments including MNIST digit generation, document layout generation, clipart abstract scene generation and tangram graphic design.
Summary: This paper presents a novel GAN framework for generating graphic layouts which consists of a set of graphic elements which are geometrically and semantically related. The generator learns a function that maps input layout ( a random set of graphic elements denoted by their classes probabilities and and geometric parameters) and outputs the new contextually refined layout. The paper also explores two choices of discriminators: (1) relation based discriminator which directly extracts the relations among different graphic elements in the parameter space, and (2) wireframe rendering discriminator which maps graphic elements to 2D wireframe images using a differentiable layer followed by a CNN for learning the discriminator. The novel GAN framework is evaluated on several datasets such as MNIST, document layout comparison and clipart abstract scene generation  
Pros:
- The paper is trying to solve an interesting problem of layout generation. While a large body of work has focussed on pixel generation, this paper focuses on graphic layouts which can have a wide range of practical applications. 
- The paper presents a novel architecture by proposing a generator that outputs a graphic layout consisting of class probabilities and polygon keypoints. They also propose a novel discriminator consisting of a differentiable layer that takes the parameters of the output layout and generates a rasterized image representing the wireframe. This is quite neat as it allows to utilize a CNN for learning a discriminator for real / fake prediction. 
- Qualitative results are shown on a wide variety of datasets - from MNIST to clipart scene generation and tangram graphic design generation. I found the clipart scene and tangram graphic design generation experiments quite neat. 
Cons:
- While the paper presents a few qualitative results, the paper is missing any form of quantitative or human evaluation on clip-art scene generation or tangram graphic design generation. 
- The paper also doesn’t report results on simple baselines for generating graphic layouts. Why not have a simple regression based baseline for predicting polygon parameters? Or compare with the approach mentioned in [1]
- Even for generating MNIST digits, the paper doesn’t report numbers on previous methods used for MNIST digit generation. 
Interestingly, only figure 4 shows results from a traditional GAN approach (DCGAN). Why not show the output on other datasets too? 
Questions / Remarks:
- Why is the input to the GAN not the desired graphic elements and pose the problem as just predicting the polygon keypoints for those graphic elements. I didn’t quite understand the motivation of choosing a random set of graphic elements and their class probabilities as input.  
    - How does this work for the case of clip-art generation for example? The input to the gan is a list of all graphic elements (boy, girl glasses, hat, sun and tree) or a subset of these?
    - It is also not clear what role the class probabilities are playing this formulation. 
- In section 3.3.2, it’s mentioned that the target image consist of C channels assuming there are C semantic classes for each element. What do you mean by each graphic element having C semantic classes? Also in the formulation discusses in this section, there is no further mention of C. I wasn’t quite clear what the purpose of C channels is then. 
- I found Figure 3b quite interesting - it would have been nice if you expanded on that experiments and the observations you made a little more. 
[1] Deep Convolutional Priors for Indoor Scene Synthesis by Wang et al