We propose a novel adversarial learning framework in this work. Existing adversarial learning methods involve two separate networks, i.e., the structured prediction models and the discriminative models, in the training. The information captured by discriminative models complements that in the structured prediction models, but few existing researches have studied on utilizing such information to improve structured prediction models at the inference stage. In this work, we propose to refine the predictions of structured prediction models by effectively integrating discriminative models into the prediction. Discriminative models are treated as energy-based models. Similar to the adversarial learning, discriminative models are trained to estimate scores which measure the quality of predicted outputs, while structured prediction models are trained to predict contrastive outputs with maximal energy scores. In this way, the gradient vanishing problem is ameliorated, and thus we are able to perform inference by following the ascent gradient directions of discriminative models to refine structured prediction models. The proposed method is able to handle a range of tasks, e.g., multi-label classification and image segmentation. Empirical results on these two tasks validate the effectiveness of our learning method.
Building on the work of (Gygli et al., 2017), this paper introduces a training algorithm for energy-based models for structured prediction. Similar to Gygli et al. (2017), they train an energy-based discriminator, which matches the energy value of structured outputs with their target values assigned by a value function.
The authors present the learning algorithm in an adversarial learning framework by describing a structured prediction model G and a discriminator D. However, it is very confusing for me to understand the proposed formulation as an adversarial framework. In an adversarial framework, at the equilibrium, G could be used as a final prediction model, however, the predicted output of G are still low quality. For example, considering Table 1, the performance of G is exactly the same as NN baseline, which suggests that only the first term of Eq. 6  participates in the training of G (because L_g(G, y*) is the exact objective of the NN baseline).
What that I can easily relate to, however, is that this training algorithm is similar to Gygli et al. (2017), but uses G to get an initial point for gradient-based inference. We want this initial point to be close to the target value Eq. 6. We use the initial value (prediction of G) and the ground truth as the matching constraints (Eq. 5) (as well as the other samples that construct Eq. 10). This actually describes why D can refine the output of G (because it looks at it as an initial point that needs refinement), but the discriminators in the other adversarial frameworks can't refine that much (since they have reached the equilibrium). I would love to hear authors comments on my concern regarding the proposed adversarial framework.
Other comments:
1) Lg is a surrogate loss, not the task-loss. Task-loss could be F1, IOU, BLEU, etc, which is the ultimate performance measure on a task. 
2) The authors refer to G as a structured prediction model but starting from Section 4, they have switched to call it a classifier, which is confusing.
3) "Gygli et al. (2017) found that the key to learning energy-based models is generating proper training data.":  Is this a general statement for every energy-based model? I understand its effect when matching values, but is it still true for other training algorithms such as structural SVM training (Belanger and McCallum, 2016)? Do you have evidence to support it?
4) "In the experiment, we adopt the fully convolutional network (FCN) (Long et al., 2015) baseline model proposed in (Gygli et al., 2017) as our segmentation network. It consists of three 5 × 5 convolutional layers and two deconvolution layer.": The text from Gygli et al. (2017) says three convolutional layers and two fully connected layers. Are you using the same architecture? If not, can you describe the architecture in more details? 
5) The qualitative results for Gygli et al. (2017) appear in https://gyglim.github.io/deep-value-net/. The reported output for DVN row is significantly worse than the segmentation results of the same horses specifically for columns 4 and 8, while the overall reported IOU in Table 2 is exactly the same. Can you describe the source of this disagreement?
6) Is having a continuous domain for value function v essential for the proposed training algorithm? 
=== After rebuttal ===
I am not convinced that the improved performance is because of the adversarial training. I trained a simple MLP and with the right amount of regularization it gets 42.0% f1 score on Bibtex, so I am not sure that the adversarial training is very essential here.