Sometimes, requiring explanability is counterproductive
Okay, I feel like a crazy person arguing against a luminary of the field, but this is so misleading as to be disingenuous and make me trust the guy less. He&#x27;s throwing causality and explainability into the same bucket and arguing against the need for the latter.<p>&gt;We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications.<p>&gt;Yes, that can be useful. Or at least reassuring.<p>&gt;But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation.<p>It goes on to argue mostly against the need for intuitive explanations, not the establishing of causal relationships.<p>&gt;Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing.<p>&gt;But we don&#x27;t actually have a &quot;causal&quot; explanation for it, though we do have an accurate mathematical model and decades of experimental evidence.<p>The physical models we have are causal ones. The intuitive abstractions like bernoulli&#x27;s principle may not work, but analysis based on navier stokes sure does. You plug your shape (cause) into the equations and see what forces (effect) occur. That&#x27;s causation.<p>&gt;You know what other life-critical phenomena we don&#x27;t have good causal explanations for?<p>&gt; The mechanism of action of many drugs (if not most of them).<p>Using an industry that&#x27;s nearly synonymous with a randomized controlled trial as a refutation for the need of a causal relationship is crazy talk. The <i>mechanism</i> may be missing, but the causal explanation is that via a series of RCTs it&#x27;s established that the drug <i>causes</i> the effects.<p>I get that half of this is trying to go against a percieved need for intuitive explanations, but it weirdly lumps causation in there.