Deep Infomax (DIM) is an unsupervised representation learning framework by maximizing the mutual information between the inputs and the outputs of an encoder, while probabilistic constraints are imposed on the outputs. In this paper, we propose Supervised Deep InfoMax (SDIM), which introduces supervised probabilistic constraints to the encoder outputs. The supervised probabilistic constraints are equivalent to a generative classifier on high-level data representations, where class conditional log-likelihoods of samples can be evaluated. Unlike other works building generative classifiers with conditional generative models, SDIMs scale on complex datasets, and can achieve comparable performance with discriminative counterparts. With SDIM, we could perform classification with rejection. Instead of always reporting a class label, SDIM only makes predictions when test samples' largest logits surpass some pre-chosen thresholds, otherwise they will be deemed as out of the data distributions, and be rejected. Our experiments show that SDIM with rejection policy can effectively reject illegal inputs including outof-distribution samples and adversarial examples.
The paper proposes a scalable approach to train generative classifiers using information maximizing representation learning, with the motivation that generative classifiers could be more robust to adversarial attacks than discriminative classifiers. An off-the-shelf mutual information maximizer (MINE, DIM) is used to learn low-dimensional representations of images. Then, class-conditioned generative models of the representations are learned avoiding full generative modeling of the images. An additional loss is used to train the generative classifier which maximizes likelihood margins. Finally, percentile-based thresholds of the class log-probabilities is proposed to be used to reject classification for out-of-manifold inputs.
The paper cites existing literature which indicates that generative classifiers might be more robust to adversarial attacks, and uses recently-proposed representation learning techniques to scale up learning generative generative classifiers. The motivation of the proposed technique is clear, and the problem itself is relevant. Similar prior work use generative modeling at the pixel-level. Generative modeling of representations is novel, afaik.
The technique is evaluated on out-of-distribution sample detection (FashionMnist->MNIST and Cifar10->SVHN) and adversarial attacks. FGSM, PGD, CW-L2 attack, deepfool.
The experiments section is not very clearly written. Some of the evaluation itself is nonstandard in which only the first example of digit 0 of MNIST is used. The paper needs to have a clearer explanation and interpretation of the results. It’s not clear what the logits in table 4 and table 6 are, and what is being shown by the comparison.
Summary: The authors propose a new technique for training generative classifiers with the aim to improve robustness to adversarial attacks and confidence on out-of-distribution samples. The method is well-motivated and explained, but the experiment section is not very clearly written and I’m not confident whether the technique represents an advancement in the state-of-the-art or not.
Misc comments:
“By adopting a generative approach p(x, y) = p(y)p(x|y), we assume that the data follows the manifold assumption: the (high-dimensional) data lies on low-dimensional manifolds corresponding to their class labels.”
This sentence seems to be making a stronger claim than is needed, which is possibly incorrect. Assuming a generative approach doesn’t require assuming the manifold assumption. 
Nit: “possible MI low-bounds”
Nit: “state-of-the-arts”
Typo: “The original image is the fist sample of class 0”