Models parameterized by deep neural networks have achieved state-of-the-art results in many domains. These models are usually trained using the maximum likelihood principle with a finite set of observations. However, training deep probabilistic models with maximum likelihood can lead to the issue we refer to as input forgetting. In deep generative latent-variable models, input forgetting corresponds to posterior collapse-a phenomenon in which the latent variables are driven independent from the observations. However input forgetting can happen even in the absence of latent variables. We attribute input forgetting in deep probabilistic models to the finite sample dilemma of maximum likelihood. We formalize this problem and propose a learning criterion-termed reflective likelihood-that explicitly prevents input forgetting. We empirically observe that the proposed criterion significantly outperforms the maximum likelihood objective when used in classification under a skewed class distribution. Furthermore, the reflective likelihood objective prevents posterior collapse when used to train stochastic autoencoders with amortized inference. For example in a neural topic modeling experiment, the reflective likelihood objective leads to better quantitative and qualitative results than the variational auto-encoder and the importance-weighted autoencoder.
Summary:
This paper proposes maximizing the “reflective likelihood,” which the authors define as: E_x E_y [log q(y|x) - \alpha log q(y)] where the expectations are taken over the data, q is the classifier, and \alpha is a weight on the log q(y) term.  The paper derives the reflective likelihood for classification models and unsupervised latent variable models.  Choices for \alpha are also discussed, and connections are made to ranking losses.  Results show superior F1 and perplexity in MNIST classification and 20NewsGroups modeling.
Pros:
I like how the paper frames the reflective likelihood as a ranking loss.  It does seem like subtracting off the marginal probability of y from the conditional likelihood should indeed ‘focus’ the model on the dependent relationship y|x.  Can this be further formalized?  I would be very interested in seeing a derivation of this kind.    
I like that the authors test under class imbalance and report F1 metrics in the experiments as it does seem the proposed method operates through better calibration.
Cons:
My biggest issue with the paper is that I find much of the discussion lacks rigor.  I followed the argument through to Equation 3, but then I became confused when the discussion turned to ‘dependence paths’: “we want our learning procedure to follow the dependence path—the subspace in Θ for which inputs and outputs are dependent. However this dependence path is unknown to us; there is nothing in Eq. 1 that guides learning to follow this dependence path instead of following Eq. 3—the independence path” (p 3).  What are these dependence paths?  Can they be defined mathematically in a way that is more direct than switching around the KLD directions in Equations 1-3?  Surely any conditional model x-->y has a ‘dependence path’ flowing from y to x, so it seems the paper is trying to make some stronger statement about the conditional structure?
Moving on to the proposed reflective likelihood in Equation 4, I could see some connections to Equations 1-3, but I’m not sure how exactly that final form was settled upon.  There seems to be a connection to maximum entropy methods?  That is,   E_x E_y [log q(y|x) - \alpha log q(y)] = E_x E_y [log q(y|x)] + \alpha E_y [ -log q(y)] \approx E_x E_y [log q(y|x)] + \alpha H[y], if we assume q(y) approximates the empirical distribution of y well.  Thus, the objective can be thought of as maximizing the traditional log model probability plus an estimate of the entropy.  As there is a long history of maximum entropy methods / classifiers, I’m surprised there were no mentions or references to this literature.  Also, I believe there might be some connections to Bayesian loss calibration / risk by viewing \alpha as a utility function (which is easy to do when it is defined to be data dependent).  I’m less sure about this connection though; see Cobb et al. (2018) (https://arxiv.org/abs/1805.03901) and its citations for references.   
The data sets used in the experiments are also somewhat dissatisfying as MNIST and 20NewsGroups are fairly easy to get high-performing models for.  I would have liked to have seen more direct analysis / simulation of what we expect from the reflective likelihood.  As I mentioned above, I suspect its really providing gains through better calibration---which the authors may recognize as F1 scores are reported and class imbalance tested---but the word ‘calibration’ is never mentioned.  More direction comparison against calibration methods such as Platt scaling would be make the experiments have better focus.  It would be great to show that this method provides good calibration directly during optimization and doesn’t need the post-hoc calibration steps that most methods require. 
Evaluation:  While the paper has some interesting ideas, they are not well defined, making the paper unready for publication.  Discussion of the connections to calibration and maximum entropy seems like a large piece missing from the paper’s argument.