We introduce contextual explanation networks (CENs)-a class of models that learn to predict by generating and leveraging intermediate explanations. CENs are deep networks that generate parameters for context-specific probabilistic graphical models which are further used for prediction and play the role of explanations. Contrary to the existing post-hoc model-explanation tools, CENs learn to predict and to explain jointly. Our approach offers two major advantages: (i) for each prediction, valid instance-specific explanations are generated with no computational overhead and (ii) prediction via explanation acts as a regularization and boosts performance in low-resource settings. We prove that local approximations to the decision boundary of our networks are consistent with the generated explanations. Our results on image and text classification and survival analysis tasks demonstrate that CENs are competitive with the state-of-the-art while offering additional insights behind each prediction, valuable for decision support.
the paper is clearly written; it works on a popular idea of combining graphical models and neural nets.
this work could benefit from differentiating more from previous literature.
one key component is interpretability, which comes from the use of graphical models.  the authors claim that the previous art directly integrate neural networks into the graphical models as components, which renders the models uninterpretable. however, it is unclear, following the same logic, why the proposed method has interpretability. after all, how to go from the context to the parameters of the graphical models is still uninterpretable. specifically, it is helpful to pinpoint what is special in this model that makes it interpretable, compared to works like Gao, Y., Archer, E. W., Paninski, L., & Cunningham, J. P. (2016). NIPS or Johnson, M., Duvenaud, D. K., Wiltschko, A., Adams, R. P., & Datta, S. R. (2016). NIPS. also, is there any methodological advancement essential to CENs? 
the other idea is to go context specific. this idea has been present in language modeling, for example, amortized embedding models like M. Rudolph, F. Ruiz, S. Athey, and D. Blei (2017). NIPS and L. Liu, F. Ruiz, S. Athey, and D. Blei.  (2017). NIPS. application to medical data is interesting. but it could be helpful for the readers to understand if the idea in this work is fundamentally different from these previous ideas from amortized inference.
a final thing. a common challenge with composing graphical models and neural networks (in interpretable or uninterpretable ways) is that the neural networks will usually eat up all the representational power. the variance captured by graphical models becomes negligible. to this end, the power of graphical models for interpretability is limited. interpretability in this case is not much different from fitting only a neural network, taking the penultimate layer to the output as "context specific features" can claim that we are composing a linear model with a neural network, and the linear model is interpretable. it would be interesting to be clear about how the authors get around this issue.