Understanding black-box machine learning models is important towards their widespread adoption. However, developing globally interpretable models that explain the behavior of the entire model is challenging. An alternative approach is to explain black-box models through explaining individual prediction using a locally interpretable model. In this paper, we propose a novel method for locally interpretable modeling -Reinforcement Learning-based Locally Interpretable Modeling (RL-LIM). RL-LIM employs reinforcement learning to select a small number of samples and distill the black-box model prediction into a low-capacity locally interpretable model. Training is guided with a reward that is obtained directly by measuring agreement of the predictions from the locally interpretable model with the black-box model. RL-LIM near-matches the overall prediction performance of black-box models while yielding human-like interpretability, and significantly outperforms state of the art locally interpretable models in terms of overall prediction performance and fidelity.
This paper studies the problem of interpreting predictions of blackbox models. In particular, they study local interpretable models, which are used to study interpretability at the level of one or a few data points. A key challenge is that in order for local models to be interpretable, they need to be simple in form and therefore lower in capacity (i.e. linear); thus, if they are trained on entire datasets they will underfit.
The aim of this work is to address this issue by learning how to select representative samples from a dataset for training local linear models to reproduce predictions of black-boxes. In particular, they propose to use RL to learn to weight instances from datasets; RL is required as they make hard (i.e. non-differentiable) decisions to select a subset of the dataset. 
This work is closely related to Ren et al. 2018 [1], which proposes to meta-learn how to weight samples in a batch so as to maximize performance on a validation set. By analogy, in this paper, the samples are taken from the entire dataset (i.e. the dataset is subsampled), while the validation loss is effectively an imitation loss formed by treating the black-box model predictions as a target. The data subsampling operation introduces the added complication of non-differentiability. The contribution of this paper is thus the use of RL for meta-learning how to subsample a larger dataset in order to maximize some validation loss. 
		
* Pros:
	* Considers an interesting dataset subsampling variant of the sample weighting meta-learning problem.
	* Novel application of meta-learning for improving locally-linear models.
	* Extensive quantitative evaluation shows that the method seems to perform better than baselines, though it might be that a differentiable approximation could do as well while being more sample efficient.
	* It is a nice result that the l1 penalty actually works well in reducing the number of samples chosen by the 
	* I found the discussion and figures presented in 4.2 to be quite nice and informative.
* Cons:
	* Given the lack of a differentiable approximation baseline, I am not entirely convinced that the use of RL is absolutely necessary/optimal.
		* I.e. if the weighting function is actually high-entropy, randomly sampling a (large) batch and weighting it might work just as well.  
	* Though there is discussion of the complexity of the overall method it would be nice to see a discussion and figures related to the sample efficiency of REINFORCE?
		* This would be strongest if given with a comparison to differentiable alternatives (mentioned above) as well.
		* This would help elucidate whether RL is optimal in this setting: fitting a linear model on more data might be cheaper learning to subsample with REINFORCE. 
	* While the sample weighting function is fast at inference time, most of the overhead comes at training time. This function needs be updated in settings where the underlying dataset changes.
	* This is a minor issue, but this pushes the burden of interpretability further up to the black-box sample weighting function. While this interpretability problem is less critical, it still exists.
* Other comments/requests:
	* While the use of RL is certainly motivated in order to solve the problem in an unbiased way, it would be nice to see a comparison to a differentiable approximation as a baseline? A few ideas:
		* Randomly sample a (possibly large batch) and learn to weight it (closely related to the straight through estimator)
		* Randomly sample a batch and apply [1]
	* Would be nice to show the sizes of datasets and how many samples end up being used for different values of lambda.
	* Would be nice to understand which samples are chosen and why. This is probably tricky to analyze, but it would be interesting to see if certain samples are often chosen, or if the weighting distribution has an interesting shape (i.e. is low or high-entropy).
I’ve given a weak accept, conditioned on being provided more evidence regarding 1) comparisons to simple differentiable alternatives, 2) sample efficiency of the RL method, and 3) basic analysis of the weighting function. 
[1] Learning to Reweight Examples for Robust Deep Learning. Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun. https://arxiv.org/abs/1803.09050