Is Artificial Intelligence Permanently Inscrutable? (2016)
If AI is impervious to explaining itself, then we&#x27;ve created something that we are unable to evaluate, aside from some kind of generic phrase like &quot;The machine is looking at a bunch of stuff, and some of those things are associated with some other things. Sometimes. So it&#x27;s guessing this is the appropriate action&quot;<p>I&#x27;m not going to get into the dystopian nature of that. Too easy. Instead, I have a very practical question: how do we know that the recommended action is doing more good than harm? If we can&#x27;t understand it, how can we tell whether it&#x27;s working or not? Use statistical sampling? We quickly end up in a spot where we have Tesla that drives into guard rail. It works for everything we&#x27;ve seen so far, given this rather fuzzy idea of what &quot;works&quot; means. That&#x27;s pretty good, right?<p>With a car, if it crashes that&#x27;s a spectacular event. People cover that edge case quickly. But what happens if the AI simply wastes an hour of ten million people&#x27;s time? Would we even know? What if some bad training data or whatnot led the AI to shift 5% of an election result because it chose certain types of information to share with them and refrained from sharing others? If <i>we</i> don&#x27;t know what it&#x27;s doing or why, would those people have any idea that they were being subtly influenced or how that happened? If this true, if we can never understand, a world full of AI &quot;helping&quot; us is a world full of billions of people being subtly influenced to act in various ways because of broad promises and reasons they could never understand.<p>Back when science fiction had a lot of evil computers, they always wanted to directly control humans. That was easy to hate and fight against. This kind of thing is just as dangerous, but I&#x27;m not even sure you need an evil computer. Which is better, a malevolent intelligence out for world domination, or several dozen poorly-trained AI recommendation engines optimizing for variables and sample instance that the coders might care a lot more about than any specific user? Dang. I ended up in dystopia-land anyway. Sorry about that.