Disentangled GANs for Controllable Generation of High-Resolution Images
Generative adversarial networks (GANs) have achieved great success at generating realistic samples. However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, for demonstrating that the controllable generation of high-resolution images is possible with sufficient supervision. More importantly, only using 5% of the labelled data significantly improves the disentanglement quality. Inspired by the observed separation of fine and coarse styles in StyleGAN, we then extend AC-StyleGAN to a new image-to-image model called FC-StyleGAN for semantic manipulation of fine-grained factors in a high-resolution image. In experiments, we show that FC-StyleGAN performs well in only controlling fine-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images. Finally, we create two new datasets -- Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.
Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets. My rating is weak reject. While the paper has a variety of contributions (most notably introducing two new datasets and modifications of StyleGAN) and interesting results (such as relatively high performance with only 5% labeled data and the disentanglement issues when controlling a subset of factors), the core contributions of new datasets and architectures are not validated or analyzed rigorously enough. 1) No prior work is used as baselines in order to compare / validate the newly introduced architectures. The authors dismiss prior work in the introduction but do not provide any direct evidence that prior work is unable to handle the datasets introduced in the paper. In order to consider acceptance based on the value of the architectures proposed in the paper, there should be some direct evidence that the proposed Style-GAN modifications AC/FC, outperform prior architectures. One particular choice which is unclear to the reviewer is why AC GAN was chosen over the Projection Discriminator of Miyato and Koyama 2018 which demonstrated significantly better results than AC GAN and was adopted by major followup work such as BigGAN. 2) Given the core contributions of new architectures for disentanglement, the lack of any results on real, non synthetic, datasets in order to validate these (for instance, CelebA) which significant prior work on disentanglement has been evaluated on is an unfortunate omission. It is unclear how the introduced architectures handle complex natural image distributions with the much more diverse sources of natural variation they contain. 3) The authors claim their method is effective in the semi-supervised setting and demonstrate this by showing relatively strong performance in a low data regime. This demonstrates the importance of the third term in the loss function, but as other work for semi-supervised learning has shown (Oliver et al 2018) care must be take to properly attribute the performance of a semi-supervised algorithm to the actual semi-supervised components and purely supervised baselines are often quite competitive. The addition of an ablation / analysis demonstrating the contribution of the second term in settings where labels are also available would strengthen the author*s claims and fully demonstrate the model is an effective semi-supervised learner. Additional comments: The datasets introduced in the paper do seem like potentially valuable contributions to the community - more discussion on the motivations behind their creation, the differences with prior work, the open difficulties / challenges, as well as the recommended evaluation protocols could add to their value. The authors could more clearly motivate the work / applications the paper is interested in and how each contribution / experiment fits in with this. Without a clear sense of the authors mission / goals with the work, it feels a bit difficult to interpret the results.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Disentangled GANs for Controllable Generation of High-Resolution Images
Generative adversarial networks (GANs) have achieved great success at generating realistic samples. However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, for demonstrating that the controllable generation of high-resolution images is possible with sufficient supervision. More importantly, only using 5% of the labelled data significantly improves the disentanglement quality. Inspired by the observed separation of fine and coarse styles in StyleGAN, we then extend AC-StyleGAN to a new image-to-image model called FC-StyleGAN for semantic manipulation of fine-grained factors in a high-resolution image. In experiments, we show that FC-StyleGAN performs well in only controlling fine-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images. Finally, we create two new datasets -- Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets. My rating is weak reject. While the paper has a variety of contributions (most notably introducing two new datasets and modifications of StyleGAN) and interesting results (such as relatively high performance with only 5% labeled data and the disentanglement issues when controlling a subset of factors), the core contributions of new datasets and architectures are not validated or analyzed rigorously enough. 1) No prior work is used as baselines in order to compare / validate the newly introduced architectures. The authors dismiss prior work in the introduction but do not provide any direct evidence that prior work is unable to handle the datasets introduced in the paper. In order to consider acceptance based on the value of the architectures proposed in the paper, there should be some direct evidence that the proposed Style-GAN modifications AC/FC, outperform prior architectures. One particular choice which is unclear to the reviewer is why AC GAN was chosen over the Projection Discriminator of Miyato and Koyama 2018 which demonstrated significantly better results than AC GAN and was adopted by major followup work such as BigGAN. 2) Given the core contributions of new architectures for disentanglement, the lack of any results on real, non synthetic, datasets in order to validate these (for instance, CelebA) which significant prior work on disentanglement has been evaluated on is an unfortunate omission. It is unclear how the introduced architectures handle complex natural image distributions with the much more diverse sources of natural variation they contain. 3) The authors claim their method is effective in the semi-supervised setting and demonstrate this by showing relatively strong performance in a low data regime. This demonstrates the importance of the third term in the loss function, but as other work for semi-supervised learning has shown (Oliver et al 2018) care must be take to properly attribute the performance of a semi-supervised algorithm to the actual semi-supervised components and purely supervised baselines are often quite competitive. The addition of an ablation / analysis demonstrating the contribution of the second term in settings where labels are also available would strengthen the author*s claims and fully demonstrate the model is an effective semi-supervised learner. Additional comments: The datasets introduced in the paper do seem like potentially valuable contributions to the community - more discussion on the motivations behind their creation, the differences with prior work, the open difficulties / challenges, as well as the recommended evaluation protocols could add to their value. The authors could more clearly motivate the work / applications the paper is interested in and how each contribution / experiment fits in with this. Without a clear sense of the authors mission / goals with the work, it feels a bit difficult to interpret the results.
assistant
Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets. My rating is weak reject. While the paper has a variety of contributions (most notably introducing two new datasets and modifications of StyleGAN) and interesting results (such as relatively high performance with only 5% labeled data and the disentanglement issues when controlling a subset of factors), the core contributions of new datasets and architectures are not validated or analyzed rigorously enough. 1) No prior work is used as baselines in order to compare / validate the newly introduced architectures. The authors dismiss prior work in the introduction but do not provide any direct evidence that prior work is unable to handle the datasets introduced in the paper. In order to consider acceptance based on the value of the architectures proposed in the paper, there should be some direct evidence that the proposed Style-GAN modifications AC/FC, outperform prior architectures. One particular choice which is unclear to the reviewer is why AC GAN was chosen over the Projection Discriminator of Miyato and Koyama 2018 which demonstrated significantly better results than AC GAN and was adopted by major followup work such as BigGAN. 2) Given the core contributions of new architectures for disentanglement, the lack of any results on real, non synthetic, datasets in order to validate these (for instance, CelebA) which significant prior work on disentanglement has been evaluated on is an unfortunate omission. It is unclear how the introduced architectures handle complex natural image distributions with the much more diverse sources of natural variation they contain. 3) The authors claim their method is effective in the semi-supervised setting and demonstrate this by showing relatively strong performance in a low data regime. This demonstrates the importance of the third term in the loss function, but as other work for semi-supervised learning has shown (Oliver et al 2018) care must be take to properly attribute the performance of a semi-supervised algorithm to the actual semi-supervised components and purely supervised baselines are often quite competitive. The addition of an ablation / analysis demonstrating the contribution of the second term in settings where labels are also available would strengthen the author*s claims and fully demonstrate the model is an effective semi-supervised learner. Additional comments: The datasets introduced in the paper do seem like potentially valuable contributions to the community - more discussion on the motivations behind their creation, the differences with prior work, the open difficulties / challenges, as well as the recommended evaluation protocols could add to their value. The authors could more clearly motivate the work / applications the paper is interested in and how each contribution / experiment fits in with this. Without a clear sense of the authors mission / goals with the work, it feels a bit difficult to interpret the results.