There has been recent interest in improving performance of simple models for multiple reasons such as interpretability, robust learning from small data, deployment in memory constrained settings as well as environmental considerations. In this paper, we propose a novel method SRatio that can utilize information from high performing complex models (viz. deep neural networks, boosted trees, random forests) to reweight a training dataset for a potentially low performing simple model such as a decision tree or a shallow network enhancing its performance. Our method also leverages the per sample hardness estimate of the simple model which is not the case with the prior works which primarily consider the complex model's confidences/predictions and is thus conceptually novel. Moreover, we generalize and formalize the concept of attaching probes to intermediate layers of a neural network to other commonly used classifiers and incorporate this into our method. The benefit of these contributions is witnessed in the experiments where on 6 UCI datasets and CIFAR-10 we outperform competitors in a majority (16 out of 27) of the cases and tie for best performance in the remaining cases. In fact, in a couple of cases, we even approach the complex model's performance. We also conduct further experiments to validate assertions and intuitively understand why our method works. Theoretically, we motivate our approach by showing that the weighted loss minimized by simple models using our weighting upper bounds the loss of the complex model.
The paper proposes a method for transferring knowledge from a complex, high-performing model to a small/lower-performing one. The approach is based on reweighting the train examples according to the ratio of confidences of the complex and simple models, and then retraining the simple model on the reweighed dataset. 
Pros:
* Some interesting ideas motivating the approach
Cons:
* An important error in the main result and various other issues (imprecise statements, lack of details) raise doubts on the theoretical contribution of the paper.
* Limited novelty. Contribution boils down to importance reweighting via a ratio of probabilities.
* Limited awareness of related work.
* Experimental framework lacks meaningful baselines and important details .
Based on the points above, I have to recommend rejection of this paper to ICLR. 
Detailed Comments:
* The claim that per-sample hardness reweighting is novel to this work is a bit of a stretch. Most reweighting-based methods for distillation use some form of this idea, though how each method estimates “hardness” varies, ranging from raw confidence scores of the teacher model (e.g., classic distillation), to scaled versions of it, to a ratio of this and the student’s score (this work).
* More generally, there’s quite a few very relevant related works which are not cited here (see list below for a non-exhaustive list).
* I believe the inequality used to prove Lemma 3.2 is not true in general. It’s easy to come up with counterexamples for which w>=1 and x>1 yet log(wx) <= w log(x) does not hold. For example, for w=2, any x in (1,2) will not satisfy this. In general, this inequality is only true if x >= w^{1/(w-1)}. I would be prepared to pass this off as a minor error, were it not for the fact that the w and x in the proof of Lemma 3.2 could quite conceivably be outside this range, i.e., rendering the inequality (and therefore the bound) invalid.
* There are various other technical aspects of the paper that are either non-rigorous or lack details. For example, Lemma 3.1 requires more information. For which y does w>= hold? Is it just for the true class y*? The proof is equally obscure, and seems to use the same problematic inequality of Lemma 3.2 (for the cross-entropy loss). 
* The statement of Lemma 3.2 is confusing. The last sentence should read loss smaller than *that* of theta*. Also, I don’t see the optimality of theta* being used anywhere. The statement could very well be phrased in terms of any two models, one of which has a lower loss than the other. 
* Assuming the complex “teacher” classifiers are very accurate (which is assumed several times throughout the paper), re-weighting training examples by max(1, p_c / p_theta) does not provide too much information beyond the ground truth labels. This is a core problem in knowledge distillation, already identified as early as Hinton et al (2015), and which is often alleviated by some form of temperature annealing or by operating on pre-softmax logits rather than final probabilities. That doesn’t seem to be the case here, so I’m quite puzzled by the “intuitive justification” provided in this work.
* The discussion in the first paragraph of pg 4 is a well known dichotomy (namely, up-weighting easy or hard examples) in the distillation/boosting/instance-reweighting literature. This discussion would really benefit from awareness of previous work discussing this duality (e.g., (Bengio et al 2009; Ren et al 2019)
* Where and how is the ordering of the “graded” classifiers used? In step (3) of Algo 1 all \zeta’s seem to be weighted equally, so I’m missing why the classifiers need to be delta-graded.
* I find it quite surprising that in almost half of the datasets in Table 1, distillation yields the exact same performance as just training the Simple Model, and sometimes even degrades the performance. Section 4.1 points out that this is actually “an equivalent model to Distillation”, but no further explanation is given, so it is hard to conjecture what might be the problem. Futhermore, as the authors point out in the introduction, Distillation alla Hinton 2015 usually assumes both teacher and student models are soft predictors (e.g., neural nets), but the models used in the UCI experiments trees and SVMs, so I wonder how exactly they did this. 
* For problems involving distillation into simple interpretable models (like trees), such as the UCI experiments, there are many reasonably strong baselines from previous work that could have been compared against (e.g. Frosst and Hinton 2017), which could have provided a more meaningful evaluation.
[1] Learning to Reweight Examples for Robust Deep Learning, Ren et al. ICML 2018
[2] Distilling a Neural Network Into a Soft Decision Tree, Frosst and Hinton, 2017
[3] Born-Again Neural Networks, Furlanello et al., ICML 2018
[4] Fidelity-Weighted Learning, Deghani et al., ICLR 2018