Recent advancements on Generative Adversarial Network (GAN) have inspired a wide range of works that generate synthetic images. However, current processes have to generate an entire image at once, and therefore resolutions are limited by memory or computational constraints. In this work, we propose COnditional COordinate GAN (COCO-GAN), which generates a specific patch of an image conditioned on a spatial position rather than the entire image at a time. The generated patches are later combined together to form a globally coherent full-image. With this process, we show that the generated image can achieve competitive quality to state-of-the-arts and the generated patches are locally smooth between consecutive neighbors. One direct implication of the COCO-GAN is that it can be applied onto any coordinate systems including the cylindrical systems which makes it feasible for generating panorama images. The fact that the patch generation process is independent to each other inspires a wide range of new applications: firstly, "Patch-Inspired Image Generation" enables us to generate the entire image based on a single patch. Secondly, "Partial-Scene Generation" allows us to generate images within a customized target region. Finally, thanks to COCO-GAN's patch generation and massive parallelism, which enables combining patches for generating a full-image with higher resolution than state-of-the-arts.
The paper describes a GAN architecture and training methodology where a generator is trained to generate "micro-" patches, being passed as input a latent vector and patch co-ordinates. Micro-patches generated for different adjacent locations with the same latent vector are combined to generate a "macro" patch. This "macro" output is trained against a discriminator that tries to label this output as real and fake, as well as predict the location of the macro patch and the value of the latent vector. The generator is trained to fool the discriminator's label, and minimize the error in the prediction of location and latent vector information.
- The paper proposes a combination of different interesting strategies. However, a major drawback of the method is that it's not clear which of these are critical to the quality of the generated output.
- Firstly, it isn't clear to me why the further breakdown of the macro patch into micro patches is useful. There appears to be no separate loss on these intermediate outputs. Surely, a DC-GAN like architecture with sufficient capacity would be as well able to generate "macro" patches. The paper needs to justify this split into micro patches with a comparison to a direct architecture that generates the macro patches (everything else being the same). Note that applications like "interpolation" of micro patches could be achieved simply by interpolating crops of the macro patch.
- As a means of simply producing high-resolution images, it appears that "PGGAN" performs better than the proposed method. Therefore, the paper doesn't clearly explain the setting when the division into patches produces a better result. It is worth noting that the idea of applying "local" critics (i.e., discriminators acting on sub-regions) isn't new (e.g., Generative Image Inpainting with Contextual Attention in CVPR 2018). What's new is the proposed method's way of achieving consistency between different regions by providing the 'co-ordinate' of the patch as input (and seeking consistency in the latent vector through a loss)---rather than applying a discriminator at a coarser level on the downsampled image. But given the poorer performance compared to PGGAN, it isn't clear that there is a an advantage to this approach.
Overall, the paper brings up some interesting ideas, but it doesn't motivate all its design choices, and doesn't make a clear argument about the settings in which the proposed method would provide an actual advantage.
===Post-rebuttal
I'm upgrading my score from 5 to 6, because some of the ablation experiments do make the paper stronger. Having said that, I still think this is a borderline paper. "Co-ordinate conditioning" is an interesting approach, but I think the paper still lacks convincing experiments for its main motivating use case: generating outputs at a resolution that won't fit in memory within a single forward pass. (This motivation wasn't clear in the initial version, but is clearer now).
The authors' displayed some high-resolution results during the rebuttal phase, but note that they haven't tuned the hyper-parameter for these (and so the results might not be the best they can be). Moreover, they scale up the sizes of their micro and macro patches so that they're still the same factor below the full image. I think a version of this paper whose main experimental focus is on high-resolution data generation, and especially, from much smaller micro-macro patches, would make a more convincing case. 
So while the paper is about at the borderline for acceptance, I do think it could be much stronger with a focus on high-resolution image experiments (which is after all, forms its motivation).