In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). It combines the purely generative INN model with an unconstrained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of a cINN are jointly optimized with a stable, maximum likelihood-based training procedure. Even though INNs and other normalizing flow models have received very little attention in the literature in contrast to GANs, we find that cINNs can achieve comparable quality, with some remarkable properties absent in cGANs, e.g. apparent immunity to mode collapse. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.
This paper proposes conditional Invertible Neural Networks (cINN), which introduces conditioning to conventional flow-based generative models. Conditioning is injected into the model via a conditional affine coupling block, which concatenates conditioning with the input to the scaling and shifting sub-networks in the coupling block. Other small modification are proposed to improve training stability at higher learning rates, including soft clamping of scaling coefficients, and Haar wavelet downsampling, which is proposed to replace the squeeze operation (pixel shuffle) that is often used in flow-based models. The invertibility of the cINN allows for style transfer by projecting images into the latent space, and then reconstructing the image with a different conditioning. The performance of the cINN is evaluated empirically on the task of colorization, where it is shown to outperform other techniques in terms of nearness to the true colours, as well as sample diversity.
Overall, I would tend to vote for accepting this paper. The base method for integrating conditioning into the flow is simple and intuitive, and additional modifications which allow for stable training at higher learning rates, such as soft clamping and Haar wavelet downsampling, appear to be very effective. Conditional models often lend themselves to a wide variety of useful applications, so I think this work could be of interest to many. 
My primary concerns with this paper are related to the comparison of image colorization methods. Specifically:
1) I would like to see a comparison to Probabilistic Image Colorization (PIC) [1], which was mentioned in the related work section but not included in the comparison of colorization models. PIC has been shown to outperform VAE-MDN in terms of diversity, so it would be good to include it. Code is available online with pretrained ImageNet models (https://github.com/ameroyer/PIC), so it should not be difficult to add.
2) Pix2pix is known to have very bad sample diversity. A more useful comparison would be to evaluate one of the newer variants of Pix2pix that emphasizes sample diversity, such as BicycleGAN [2] or MSGAN [3]. Code is also available for each of these models (https://github.com/junyanz/BicycleGAN, https://github.com/HelenMao/MSGAN), although you would need to train models from scratch.
3) Pixel-wise metrics such as MSE are bad at measuring perceptual similarity. While these pixel-wise metrics are still useful for comparison to prior work, better metrics are available for evaluating image colorization. I would recommend the use of Learned Perceptual Image Patch Similarity (LPIPS) [4] in place of pixel-wise distance measures for evaluating image similarity and diversity.
Things to improve the paper that did not impact the score:
5) I was somewhat disappointed by how little attention was spent on the Haar wavelet downsampling method. It seems like a very neat idea, but it is only briefly explored in the ablation study. It would be nice to include a more in-depth study of how it compares to the conventional pixel shuffle downsampling, perhaps in terms of stability with learning rates, and final model performance. 
6) There is some potentially related work on conditional adversarial generative flows [5] that could be added to the literature review if deemed relevant enough.
References:
[1] Amelie Royer, Alexander Kolesnikov, and Christoph H. Lampert. Probabilistic image colorization. In British Machine Vision Conference (BMVC), 2017.
[2] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.
[3] Mao, Qi, et al. "Mode seeking generative adversarial networks for diverse image synthesis." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[4] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[5] Liu, Rui, et al. "Conditional Adversarial Generative Flow for Controllable Image Synthesis." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
### Post-Rebuttal Comments ###
Thank you for including the PIC results. I did not realize that they would take so long to run, but I think it important to include comparison with the current SOTA methods so that we have some reference. It is also nice to see the addition of the LPIPS metric. My overall opinion of the paper is not much changed from before, so I will retain the same score.