Interpreting Neural Network Judgments via Minimal, Stable, and Symbolic Corrections
We present a new algorithm to generate minimal, stable, and symbolic corrections to an input that will cause a neural network with ReLU activations to change its output. We argue that such a correction is a useful way to provide feedback to a user when the network's output is different from a desired output. Our algorithm generates such a correction by solving a series of linear constraint satisfaction problems. The technique is evaluated on three neural network models: one predicting whether an applicant will pay a mortgage, one predicting whether a first-order theorem can be proved efficiently by a solver using certain heuristics, and the final one judging whether a drawing is an accurate rendition of a canonical drawing of a cat.
This work proposes a novel method that can potentially provide actionable insight to the user when a neural network makes a less than favorable decision.  The paper is interesting in that it provides stable and hence potentially actionable insight that can help the target user change an undesired outcome in the future. The work focuses on asymmetric insight in the sense that insight or suggestions are provided only when the classification is for a certain class. So it is mainly applicable to specific kind of binary classification problems where being classified into one class is more undesirable and requires justification. Some hand wavy arguments are provided in the supplement for extension to multiple classes (one vs all), however it would be good to see experiments on those in practice as it is not at all obvious how the solution extends when you have more than one undesirable class.  Technically, they solve a large linear optimization problem to obtain the region of interest based on which the insight is conveyed based on minimal perturbation. Although, they provide some tricks to scale their approach the method seems applicable mainly to smallish or medium sized neural networks (symbolic correction with 8 features is not really that high dimensional). More large scale experiments need to be conducted to justify the solutions they propose.  The related work section also is kind of thin. Relevant works on interpretability such as S Lundberg et. al. 2017, Koh et. al. 2017, Kindermans 2018 etc. are not referred to. Also a very relevant work on contrastive explanations which looks for minimal perturbations Dhurandhar et. al. 2018 is not discussed.  Focus is on actionability however, its possible that the features they select may not be actionable for every person or instance. For example, interest rate is out of control of the individual or property type is something the individual may be inflexible about depending on his intent (a person wanting to buy a commercial building may not be happy with a recommendation telling to buy a residential condo even if he can afford it as he wants to start a business). So a dicussion with such constraints would be important to have.  Minor comment:  The text refers to some green lines in figure 1c, but I fail to see any.  Post Rebuttal:  I thank the authors for their response. I still think this is a borderline paper based on my comments.