Learning What and Where to Draw
Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers. While existing models can synthesize images based on global constraints such as a class label or caption, they do not provide control over pose or object location. We propose a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location. We show high-quality 128 × 128 image synthesis on the Caltech-UCSD Birds dataset, conditioned on both informal text descriptions and also object location. Our system exposes control over both the bounding box around the bird and its constituent parts. By modeling the conditional distributions over part locations, our system also enables conditioning on arbitrary subsets of parts (e.g. only the beak and tail), yielding an efficient interface for picking part locations.
The core idea is to take advantage of labeled data which associates images with captions as well as keypoint and bounding box information in order to train conditional GANs which generate an image given the other information. The other important idea is to train another GAN to generate the missing conditioning variables, given any subset of the others (at least this was done for the keypoints, but I presume that the same principle could be applied elsewhere). A sophisticated convolutional architecture is proposed in both the generator and discriminator parts in order to take advantage of the special meaning of the conditioning variables (bounding box and keypoints).
This is a very interesting paper with really impressive results!  One thing I would have liked to see more is a "deconstruction" of the different elements in the proposed approach, to see which ingredients matter more. Of course, part of the difficulty in such an exercise is the current lack of a quantitative evaluation procedure for GANs (or other likelihood-free generative models).  Something that concerns me is the complexity of the architecture and training procedure used in these experiments. As a condition for accepting this paper, I would like the authors to confirm that they will post their code for running these experiments. Otherwise, I feel that it would be very difficult to reproduce the experiments (or a long appendix would need to be added to clarify all the experimental details). I will raise my "clarity" rating if the authors confirm that they will publish their code.  For the same reason, I would expect that the modified dataset used for the MHP experiment be released publically.  There appear to be strange "repeated patterns" or textures showing up in some of the generated images, like in fig 8 and fig 4. These seem to be artifacts and do not correspond to the training distribution. Do you have any clue about what is going on?   Minor comments and requested clarifications:  eqn between lines 99 and 100: this is not the objective typically used to actually train GANs (please clarify, mention the actual objective).  sec. 3.2: why not optimize directly \phi and \Phi wrt the GAN objective? Is there any fine-tuning of these functions with the GAN objective? If so, does it make a difference?  Fig 2 and 3. Clarify what non-linearities are used in hidden layers and in output layers. I am assuming rectifiers for the former and tanh for the latter, but please clarify.  line 152: how was the text embedding t obtained? Like in sec 4.1?  line 165: I suspect that the T should be a K.  line 178: "a denoising autoencoder over keypoints would not suffice" --> actually this is probably not true, since one could run a Markov chain associated with the denoising autoencoder in order to sample from it (see "Generalized denoising auto-encoders as generative models" NIPS 2013).  line 189: how are (x,y) set in k for unobserved keypoints?  line 210: is this pre-training the same as for CUB?  line 214: why keep the # of averaged captions always the same at training time and variable at test time? I would vary that # at training time too.  line 232+: typo "an" should be "and"