Published as a conference paper at ICLR 2018 LEARNING APPROXIMATE INFERENCE NETWORKS FOR STRUCTURED PREDICTION
ABSTRACT Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use neural network architectures to define energy functions that can capture arbitrary dependencies among parts of structured outputs. Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables and then optimizing the energy with respect to them. We replace this use of gradient descent with a neural network trained to approximate structured argmax inference. This "inference network" outputs continuous values that we treat as the output structure. We develop large-margin training criteria for joint training of the structured energy function and inference network. On multi-label classification we report speed-ups of 10-60x compared to (Belanger et al., 2017) while also improving accuracy. For sequence labeling with simple structured energies, our approach performs comparably to exact inference while being much faster at test time. We then demonstrate improved accuracy by augmenting the energy with a "label language model" that scores entire output label sequences, showing it can improve handling of long-distance dependencies in part-of-speech tagging. Finally, we show how inference networks can replace dynamic programming for test-time inference in conditional random fields, suggestive for their general use for fast inference in structured settings.
= Quality = 
Overall, the authors do a good job of placing their work in the context of related research, and employ a variety of non-trivial technical details to get their methods to work well. 
= Clarity = 
Overall, the exposition regarding the method is good. I found the setup for the sequence tagging experiments confusing, tough. See more comments below.
= Originality / Significance = 
The paper presents a clever idea that could help make SPENs more practical. The paper's results also suggest that we should be thinking more broadly about how to using complicated structured distributions as teachers for model compression.
= Major Comment =
I'm concerned by the quality of your results and the overall setup of your experiments. In particular, the principal contribution of the sequence tagging experiments seems top be different than what is advertised earlier on in the paper. 
Most of your empirical success is obtained by taking a pretrained CRF energy function and using this as a teacher model to train a feed-forward inference network. You have have very few experiments using a SPEN energy function parametrization that doesn't correspond to a CRF, even though you could have used an arbitrary convnet, RNN, etc. The one exception is when you use the tag language model. This is a good idea, but it is pretrained, not trained using the saddle-point objective you introduce. In fact, you don't have any results demonstrating that the saddle-point approach is better than simpler alternatives.
It seems that you could have written a very different paper about model compression with CRFs that would have been very interesting and you could've have used many of the same experiments. It's unclear why SPENs are so important. The idea of amortizing inference is perhaps more general. My recommendation is that you either rebrand the paper to be more about general methods for amortizing structured prediction inference using model compression or do more fine-grained experiments with SPENs that demonstrate empirical gains that leverage their flexible deep-network-based energy functions.
= Minor Comments = 
* You should mention 'Energy Based GANs"
* I don't understand "This approach performs backpropagation through each step of gradient descent, permitting more stable training but also evidently more overfitting." Why would it overfit more? Simply because training was more stable? Couldn't you prevent overfitting by regularizing more?
* You spend too much space talking about specific hyperparameter ranges, etc. This should be moved to the appendix. You should also add a short summary of the TLM architecture to the main paper body.
* Regarding your footnote discussing using a positive vs. negative sign on the entropy regularization term, I recommend checking out "Regularizing neural networks by penalizing confident output distributions."
* You should add citations for the statement "In these and related settings, gradient descent has started to be replaced by inference networks."
* I didn't find Table 1 particularly illuminating. All of the approaches seem to perform about the same. What conclusions should I make from it?
* Why not use KL divergence as your \Delta function?
* Why are the results in Table 5 on the dev data?
* I was confused by Table 4. First of all, it took me a very long time to figure out that the middle block of results corresponds to taking a pretrained CRF energy and amortizing inference by training an inference network. This idea of training with a standard loss (conditional log lik.) and then amortizing inference post-hoc was not explicitly introduced as an alternative to the saddle point objective you put forth earlier in the paper. Second, I was very surprised that the inference network outperformed Viterbi (89.7 vs. 89.1 for the same CRF energy). Why is this?
* I'm confused by the difference between Table 6 and Table 4? Why not just include the TLM results in Table 4?