Stop Explaining Black Box Models for High Stakes Decisions
&quot;sparsity is a useful measure of interpretability, since humans can handle at most 7±2 cognitive entities at once &quot;<p>If you are going to limit models to 9 features, or 9 combinations of features, or 9 rules, models are not going to work as well. This just seems like a very weak argument to introduce.<p>&quot;A black box model is either a function that is too complicated for any human to comprehend, or a function that is proprietary&quot;
These seem like two very different things. A false equivalence is then made with many arguments against proprietary models being made as if they apply to complex models.<p>&quot;It is a myth that there is necessarily a trade-off between accuracy and interpretability....However, this is often not true&quot;
It&#x27;s a bit of a straw man to suggest that there is &quot;necessarily&quot; a trade-off, as there are surely cases where there is not. One could say that there is &quot;often&quot; a trade-off between accuracy and interpretability. I don&#x27;t think there are many people out there with the naive view that one should never do feature engineering.<p>&quot;If the explanation was completely faithful to what the original model computes, the explanation would equal the original model&quot;
This is just nonsensical to me. The idea is not to be completely faithful, but to raise things up to another level of abstraction. The series of videos from Wired comes to mind where a concept is explained at multiple levels. <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=OWJCfOvochA" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=OWJCfOvochA</a><p>&quot;Black box models are often not compatible in situations where information outside the database needs to be combined with a risk assessment.&quot;
This is absolutely untrue, depending on one&#x27;s definition of often. The output of a model, whether it falls into this incorrect definition of a model or not, can be treated as a feature in another model. Ensemble learning exists.<p>COMPAS is the punching bag, but no one seems to know what it is. I haven&#x27;t seen the evidence that its performance is equal to three if statements. It certainly doesn&#x27;t have anything to say about machine learning in general, as it is set of expert designed rules. So, it is actually the kind of algorithm the author favors, except proprietary.<p>&quot;typographical errors seem to be common in computing COMPAS.... This, unfortunately, is a drawback of using black box models&quot;
Unclear why typographical errors only affect black box models.<p>The BreezoMeter case is not clear evidence of anything broader either. It is unclear whether the one error noted out of millions of predictions is from bad source data. Stretching this to concern any sort of proprietary prediction, such as mortgage ratings, is a stretch that doesn&#x27;t really tell us anything.<p>&quot;Solving constrained problems is generally harder than solving unconstrained problems.&quot;
This doesn&#x27;t make sense to me at all. All evidence is that ML works better on constrained problems.<p>The idea that CORELS is somehow better, even if it comes up with a ruleset of millions of rules, doesn&#x27;t make sense. The proposed workaround for this &quot;the model would contain an additional term only if this additional term reduced the error by at least 1%&quot; could result in the failure to create a model if no one term provided 1% on its own.<p>Scoring systems are useful, but it&#x27;s like a single layer perceptron in the example provided. You need to consider combinations of factors to see their impact. A high X is bad, unless Y is also high and Z is is low.<p>The fundamental problem here is trying to limit the power of the algorithm to the power of the human mind. From the very beginning we have used computers to do things that are difficult or impossible for us to do. In some cases the answers were provably correct, but we have now reached a point where computer generated proofs are accepted. The &quot;oracle&quot; mode of computing will compute an answer for us, but we ultimately have to choose whether or not to accept it on the basis of the evidence, much like we do with the opinion of an expert. Simple techniques such as providing one&#x27;s own test data set, and the kind of analysis done by Tetlock around Superforecasters, can go a long way to building that understanding of accuracy of predictions, such that we have a guideline for evaluating algorithms that are beyond our ability to understand.