We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks. This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data. This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree. The proposed method is conceptually simpler than joint convolutional neural network -conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials. Experimental results demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation.
This paper presents an a particular architecture for conditional discriminators in the cGAN framework. Different to the conventional approach of concatenating the conditioning information to the input, the authors propose to process them separately with two distinct convolutional networks fusing (by element-wise addition) intermediate features of the conditioning branch into the input branch at each layer.
Pros:
+ The writing is mostly clear and easy to follow.
+ I feel that exploring better conditioning strategies is an important direction. Quite often the discriminator discards additional inputs if no special measures against this behaviour are taken.
+ The proposed method seem to outperform the baselines
Cons:
- I’m generally not excited about the architecture as it seems a slight variation of the existing methods. See, for example, the PixelCNN paper [van den Oord et al., 2016] and FiLM [Perez et al., 2017].
- Theoretical justification of the approach is quite weak. The paper shows that the proposed fusion method may result in higher activation values (in case of the ReLU non-linearity, other cases are not considered at all) but this is not linked properly to the performance of the entire system. Paragraph 3 of section 3.1 (sentence 3 and onward) seems to contain a theoretical claim which is never proved.
- It seems that the authors never compare their results with the state-of-the-art. The narrative would be much more convincing if the proposed way of conditioning yielded superior performance compared to the existing systems. From the paper it’s not clear how bad/good the baselines are.
Notes/questions:
* Section 3.1, paragraph 1: Needs to be rephrased. It’s not totally clear what the authors mean here.
* Section 3.1, paragraph 4: “We observed that the fusion …” -  Could you elaborate on this? I think you should give a more detailed explanation with examples because it’s hard to guess what those “important features” are by looking at the figure.
* Figure 4: I would really want to see the result of the projection discriminator as it seems to be quite strong according to the tables. The second row of last column (which is the result of the proposed system) suspiciously resembles the ground-truth - is it a mistake?
* Figure 5: It seems that all the experiments have not been run until convergence. I’m wondering if the difference in performance is going to be as significant when the model are trained fully.
In my opinion, the proposed method is neither sufficiently novel nor justified properly. On top of that, the experimental section is not particularly convincing. Therefore, I would not recommend the paper in its present form for acceptance.