Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation
Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss. However, we reveal that this training recipe shared by almost all existing methods causes one critical side effect: lack of diversity in output samples. In order to accomplish both training stability and multimodal output generation, we propose novel training schemes with a new set of losses named moment reconstruction losses that simply replace the reconstruction loss. We show that our approach is applicable to any conditional generation tasks by performing thorough experiments on image-to-image translation, super-resolution and image inpainting using Cityscapes and CelebA dataset. Quantitative evaluations also confirm that our methods achieve a great diversity in outputs while retaining or even improving the visual fidelity of generated samples.
 The paper describes an alternative to L1/L2 errors (wrt output and one ground-truth example) that are used to augment adversarial losses when training conditional GANs. While these augmented losses are often needed to stabilize and guide GAN training, the authors argue that they also bias the optimization of the generator towards mode collapse. To address this, the method proposes two kinds of alternate losses--both of which essentially generate multiple sample outputs from the same input, fit these with a Gaussian distribution by computing the generating sample mean and variance, and try to maximize the likelihood of the true training output under this distribution. The paper provides theoretical and empirical analysis to show that the proposed approach leads to generators that produce samples that are both diverse and high-quality. I think this is a good paper and solves an important problem---where one usually had to sacrifice diversity to obtain stable training by adding a reconstruction loss. I recommend acceptance. An interesting ablation experiment might be to see what happens when one no longer includes the GAN loss and trains only with the MLMM or MCMLE losses, and compare this to training with only the L1/L2 losses. The other thing I*d like the authors to comment on are the potential shortcomings of using a simple un-correlated Gaussian to model the sample distributions. It seems that such a distribution may not capture the fact that multiple dimensions of the output (i.e., multiple pixel intensities) are not independent conditioned on the input. Perhaps, it may be worth exploring whether Gaussians with general co-variance matrices, or independent in some de-correlated space (learned from say simply the set of outputs) may increase the efficacy of these losses. ====Post-rebuttal I*ve read the other reviews and retain my positive impression of the paper. I also appreciate that the authors have conducted additional experiments based on my (non-binding) suggestions---and the results are indeed interesting. I am upgrading my score accordingly.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Harmonizing Maximum Likelihood with GANs for Multimodal Conditional Generation
Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss. However, we reveal that this training recipe shared by almost all existing methods causes one critical side effect: lack of diversity in output samples. In order to accomplish both training stability and multimodal output generation, we propose novel training schemes with a new set of losses named moment reconstruction losses that simply replace the reconstruction loss. We show that our approach is applicable to any conditional generation tasks by performing thorough experiments on image-to-image translation, super-resolution and image inpainting using Cityscapes and CelebA dataset. Quantitative evaluations also confirm that our methods achieve a great diversity in outputs while retaining or even improving the visual fidelity of generated samples.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper describes an alternative to L1/L2 errors (wrt output and one ground-truth example) that are used to augment adversarial losses when training conditional GANs. While these augmented losses are often needed to stabilize and guide GAN training, the authors argue that they also bias the optimization of the generator towards mode collapse. To address this, the method proposes two kinds of alternate losses--both of which essentially generate multiple sample outputs from the same input, fit these with a Gaussian distribution by computing the generating sample mean and variance, and try to maximize the likelihood of the true training output under this distribution. The paper provides theoretical and empirical analysis to show that the proposed approach leads to generators that produce samples that are both diverse and high-quality. I think this is a good paper and solves an important problem---where one usually had to sacrifice diversity to obtain stable training by adding a reconstruction loss. I recommend acceptance. An interesting ablation experiment might be to see what happens when one no longer includes the GAN loss and trains only with the MLMM or MCMLE losses, and compare this to training with only the L1/L2 losses. The other thing I*d like the authors to comment on are the potential shortcomings of using a simple un-correlated Gaussian to model the sample distributions. It seems that such a distribution may not capture the fact that multiple dimensions of the output (i.e., multiple pixel intensities) are not independent conditioned on the input. Perhaps, it may be worth exploring whether Gaussians with general co-variance matrices, or independent in some de-correlated space (learned from say simply the set of outputs) may increase the efficacy of these losses. ====Post-rebuttal I*ve read the other reviews and retain my positive impression of the paper. I also appreciate that the authors have conducted additional experiments based on my (non-binding) suggestions---and the results are indeed interesting. I am upgrading my score accordingly.
assistant
 The paper describes an alternative to L1/L2 errors (wrt output and one ground-truth example) that are used to augment adversarial losses when training conditional GANs. While these augmented losses are often needed to stabilize and guide GAN training, the authors argue that they also bias the optimization of the generator towards mode collapse. To address this, the method proposes two kinds of alternate losses--both of which essentially generate multiple sample outputs from the same input, fit these with a Gaussian distribution by computing the generating sample mean and variance, and try to maximize the likelihood of the true training output under this distribution. The paper provides theoretical and empirical analysis to show that the proposed approach leads to generators that produce samples that are both diverse and high-quality. I think this is a good paper and solves an important problem---where one usually had to sacrifice diversity to obtain stable training by adding a reconstruction loss. I recommend acceptance. An interesting ablation experiment might be to see what happens when one no longer includes the GAN loss and trains only with the MLMM or MCMLE losses, and compare this to training with only the L1/L2 losses. The other thing I*d like the authors to comment on are the potential shortcomings of using a simple un-correlated Gaussian to model the sample distributions. It seems that such a distribution may not capture the fact that multiple dimensions of the output (i.e., multiple pixel intensities) are not independent conditioned on the input. Perhaps, it may be worth exploring whether Gaussians with general co-variance matrices, or independent in some de-correlated space (learned from say simply the set of outputs) may increase the efficacy of these losses. ====Post-rebuttal I*ve read the other reviews and retain my positive impression of the paper. I also appreciate that the authors have conducted additional experiments based on my (non-binding) suggestions---and the results are indeed interesting. I am upgrading my score accordingly.