We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x, y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may be used and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, and out-of-distribution detection while also enabling our models to generate samples rivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and present an approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-art in both generative and discriminative learning within one hybrid model.
This work is an attempt to bridge the gap between discriminative models, which currently obtain the state of the art on most classification problems, and generative models, which (through a model of the marginal p(x)) have the potential to shine on many tasks beyond generalization to a hold-out set with minimal shift in distributions: out of distribution detection, better generalization out of distribution, unsupervised learning etc.
While much of the current work is related to normalizing flows / invertible neural networks, the authors here propose a quite simple but appealing method: A standard neural classifier is taken and the softmax is layer chopped off and replaced by an energy based model, which models the joint probability p(x,y) instead of the posterior p(y|x). The advantage is an additional degree of freedom in the scale of the logit vector, which is would have been otherwise normalized by the softmax layer and now can now model the data distribution. The downside is the loss in ease of training. Whereas (discriminative) deep networks can be easily trained by gradient descent on a cross-entropy objective, the partition function in the energy model makes this un tractable. This is addressed through sampling, similar to (Welling & Teh, 2011).
One of the biggest achievements reported by the authors is that the performance on discriminative tasks is not hurt (much) by adding the generative model. There is only a 3 point gap between Wide-ResNet and the proposed model (92.9% vs. 95.8%) … but on what dataset? 3 datasets are mentioned in the experimental section, but table 1 does not mention on which datasets the accuracy is reported. My guess is that this is a mean or mixture, since GEM performances of 96.7% and 72.2% are reported for SVHN and CIFAR10, respectively, but this should be made clearer. 	
On out of distribution detection, could the authors comment on the histograms in table 2, in particular the difference between the new measure (AM JEM) compared to JEM log p(x) on CelebA? The proposed measure does not seem to fare well here.
Although the method does not outperform the gold standard of adversarial training, I found the models robustness to adversarial examples quite appealing, given that it was not trained for this objective (which also means that it does not require an adaptation to a norm). 
I was very impressed by Figure 6 showing distal adversarial initialized from random images, showing pretty clear images of the modelled class. The modelled variations require more investigation to verify whether we have a collapse for each class, but the results look very promising.
The paper is well written and easy to understand. A couple of details on the training procedure are missing in the experimental part. It is stated that, both, p(y|x) and the generative part p(x), are optimized, but how are these exactly integrated? Given the difficult in training this model reported in the paper, this seems to be particularly important.
I also appreciated the description of the limitations of the algorithm, and the details in the appendix (ICLR should go back to unlimited paper lengths, btw.).
More information on complexity (training times etc.) should also be helpful.