Human-in-the-Loop Interpretability Prior
We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.
Overview: This paper introduces a bayesian framework for incorporating human feedback into the construction of interpretable models.  The basic idea is to define a prior distribution over models that reflects the interpretability of a model M.  In this paper, the (expected) time for a user to predict the label of points assigned by model M is used.  Evaluating the prior is expensive; each call to p(M) requires a user study measuring HIS(x, M) for a sample of x values.  To reduce expense, the authors propose to only query models that have a sufficiently high likelihood (high accuracy) given the observed data.  The accompanying evaluation study shows that common proxies for interpretability (e.g., things like tree size) disagree among one another.  Furthermore, which proxy is best aligned with the human-assessed interpretability prior differs across use cases (data sets).  Comments:   Overall I found this to be a very interesting paper that’s largely very well written.  The idea of using user studies to elicit the relevant notion of “interpretability” is an interesting one.  Furthermore, it opens the door to follow-up studies that are specialized to specific downstream tasks.  (This sort of future direction could be mentioned in the discussion).  I did get a bit lost in sections of the experimental results.  I ask that the authors clarify the evaluation presented in Section 6.  More specific comments are provided below.    - In the introduction you write that interpretability depends on the “subjective experience of users.”  More than that, it depends on the purpose for which an explanation is being desired.  Assessing whether a model is fit-for-purpose would entail defining a specific task, which as you state is not something you do in this paper.  Nevertheless I think it’s an important part of framing the general problem.   - 3.3 prior for arbitrary models:  I am not convinced by the idea that the interpretability of local proxies reflect the interpretability of the underlying arbitrary non-interpretable model.  Have you considered framing this part of the analysis by saying there are two components: the base model (M) and the explanation model (e.g., local-proxy(M, x)).  In this setup one could search over both explanation models and base models.  Plausibly, an explanation model that works well for some M might work poorly for others.    - 4.3, Appendix B: Similarity kernel.  It seems like the similarity kernel is being defined in a way that two models that differ significantly in terms of their interpretability may nevertheless appear similar.  Consider the setting where our model class is regression models.  Correct me if I am wrong here, but I believe the proposed strategy for this class would amount to an RBF kernel between the estimated coefficient vectors.  This would say that a sparse model with coefficient vector (w_1, …, w_k, 0, …, 0) is similar to (w_1, …, w_k, e_1, …, e_m) where the e_j are small coefficients.  The same model could be deemed less similar to (w1, …, w_{k-1}, 0, w_k, 0, …, 0), depending on the choice of w_k.    Yet from a structural and interpretability standpoint, the latter model, being sparse and just weighting a different coefficient, seems more similar than the first (which might have a large number of small non-zero entries).  I suppose you can argue that if the e_j are all really small then they don’t really affect the model predictions much.  But I’m still not sure that the similarity kernels are capturing quite the right type of similarity.    - Section 6: I’m not sure I correctly understand lines 222 - 229.  What is meant by “parameter setting”?  Are these the parameters of your similarity kernel?  Is the following interpretation correct: You have 500 models, each of which is being assessed by one of k proxies (proxy measures of interpretability, such as sparsity).  To get an x for Figure 2(a) you look at the best model for each of the k proxies, and look at where it’s ranked by each of the remaining (applicable) k-1 proxies.  In the worst case, the best model for proxy A could be the worst (rank 500) model for proxy B.  Is this right?  - Section 6: Computing risk proxy on small sample.  Again, I’m not sure I correctly understand lines 233 - 241.  How are you defining “the right proxy” here?  Could you please further clarify the setup for this evaluation?  - Appendix Figure 5: The y-axis scale is too large.  No values exceed 20, but the axis goes up to 40.  - There are a handful of typos throughout the paper.  (E.g., line 254-255 “features non-zero features”)  Unfortunately I didn’t note all of them down as I was reading.  Please give the manuscript a couple of editing passes to weed these out.