Answers.Body: <blockquote>
    <p>Why do we need explainable AI?
    ... why we need to know "how does its intelligence work?"</p>
  </blockquote>
  
  <p>Because anyone with access to the equipment, enough skill, and enough time, can force the system to make a decision that is unexpected. The owner of the equipment, or 3rd parties, relying on the decision without an explanation as to why it is correct would be at a disadvantage.</p>
  
  <p>Examples - Someone <strong><em>might</em></strong> discover:</p>
  
  <ul>
  <li><p>People whom are named John Smith and request heart surgery on: Tuesday mornings, Wednesday afternoons, or Fridays on odd days and months have a 90% chance of moving to the front of the line.</p></li>
  <li><p>Couples whom have the male's last name an odd letter in the first half of the alphabet and apply for a loan with a spouse whose first name begins with a letter from the beginning of the alphabet are 40% more likely to receive the loan if they have fewer than 5 bad entries in their credit history.</p></li>
  <li><p>etc.</p></li>
  </ul>
  
  <p>Notice that the above examples <em>ought not</em> to be determining factors in regards to the question being asked, yet it's possible for an adversary (with their own equipment, or knowledge of the algorithm) to exploit it.</p>
  
  <p><strong>Source papers</strong>:</p>
  
  <ul>
  <li><p>"<a href="https://arxiv.org/abs/1908.08705" rel="noreferrer">AdvHat: Real-world adversarial attack on ArcFace Face ID system</a>" (Aug 23 2019) by Stepan Komkov and Aleksandr Petiushko</p>
  
  <ul>
  <li>Creating a sticker and placing it on your hat fools facial recognition system.</li>
  </ul></li>
  <li><p>"<a href="https://arxiv.org/abs/1906.03444" rel="noreferrer">Defending against Adversarial Attacks through Resilient Feature Regeneration</a>" (Jun 8 2019), by Tejas Borkar, Felix Heide, and Lina Karam</p>
  
  <ul>
  <li><blockquote>
    <p>"Deep neural network (DNN) predictions have been shown to be vulnerable to carefully crafted adversarial perturbations. Specifically, so-called universal adversarial perturbations are image-agnostic perturbations that can be added to any image and can fool a target network into making erroneous predictions. Departing from existing adversarial defense strategies, which work in the image domain, we present a novel defense which operates in the DNN feature domain and effectively defends against such universal adversarial attacks. Our approach identifies pre-trained convolutional features that are most vulnerable to adversarial noise and deploys defender units which transform (regenerate) these DNN filter activations into noise-resilient features, guarding against unseen adversarial perturbations.".</p>
  </blockquote></li>
  </ul></li>
  <li><p>"<a href="https://arxiv.org/abs/1710.08864" rel="noreferrer">One pixel attack for fooling deep neural networks</a>" (May 3 2019), by Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi</p>
  
  <ul>
  <li>Altering <strong>one pixel</strong> can cause these errors:</li>
  </ul>
  
  <blockquote>
    <p><a href="https://i.sstatic.net/X9yf8.jpg" rel="noreferrer"><img src="https://i.sstatic.net/X9yf8.jpg" alt="Figure 1"></a><br>
    <sub>Fig. 1. One-pixel attacks created with the proposed algorithm that successfully fooled three types of DNNs trained on CIFAR-10 dataset: The All convolutional network (AllConv), Network in network (NiN) and VGG. The original class labels are in black color while the target class labels and the corresponding confidence are given below.</sub></p>
    
    <p>&nbsp;</p>
    
    <p><a href="https://i.sstatic.net/6RdaP.jpg" rel="noreferrer"><img src="https://i.sstatic.net/6RdaP.jpg" alt="Figure 2"></a><br>
    <sub>Fig. 2. One-pixel attacks on ImageNet dataset where the modified pixels are highlighted with red circles. The original class labels are in black color while the target class labels and their corresponding confidence are given below.</sub></p>
  </blockquote></li>
  </ul>
  
  <p>Without an explanation as to how and why a decision is arrived at the decision can't be absolutely relied upon.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T23:20:43.667Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105229/discussion-on-answer-by-rob-why-do-we-need-explainable-ai).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T23:19:56.493Z
Answers.LastActivityDate: 2019-09-02T23:19:56.493Z
Answers.UserDisplayName: Rob
Answers.Body: <p>If you're a bank, hospital or any other entity that uses predictive analytics to make a decision about actions that have huge impact on people's lives, you would not make important decisions just because Gradient Boosted trees told you to do so. Firstly, because it's risky and the underlying model might be wrong and, secondly, because in some cases it is illegal - see <a href="https://en.wikipedia.org/wiki/Right_to_explanation" rel="noreferrer">Right to explanation</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T13:23:58.630Z
Answers.LastActivityDate: 2019-09-01T19:42:49.803Z
Answers.LastEditDate: 2019-09-01T19:42:49.803Z
Answers.UserDisplayName: Tomasz Bartkowiak
Answers.Body: <p>Another reason: In the future, AI might be used for tasks that are not possible to be understood by human beings, by understanding how given AI algorithm works on that problem we might understand the nature of the given phenomenon.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-31T15:29:44.573Z
Answers.LastActivityDate: 2019-08-31T15:35:01.863Z
Answers.LastEditDate: 2019-08-31T15:35:01.863Z
Answers.UserDisplayName: GKozinski
Answers.Body: <p>The answer to this is incredibly simple. If you are a bank executive one day you may need to stand up in court and explain why your AI denied mortgages to all these people... who just happen to share some protected characteristic under anti-discrimination legislation. The judge will not be happy if you handwave the question away mumbling something about algorithms. Or worse, why did this car/plane crash and how will you prevent it next time. </p>
  
  <p>This is the major blocker to more widespread adoption of AI in many industries.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-04T16:19:24.513Z
Answers.Text: "The answer" is overstating your case. I would upvote if you said "One answer ..."
Answers.UserDisplayName: John Coleman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T11:05:46.330Z
Answers.LastActivityDate: 2019-09-02T11:05:46.330Z
Answers.UserDisplayName: Gaius
Answers.Body: <p>Explainable AI is often desirable because</p>
  
  <ol>
  <li><p>AI (in particular, artificial neural networks) can <a href="https://en.wikipedia.org/wiki/Catastrophic_failure" rel="noreferrer">catastrophically fail</a> to do their intended job. More specifically, it can be hacked or attacked with <a href="https://arxiv.org/pdf/1312.6199.pdf" rel="noreferrer">adversarial examples</a> or it can take unexpected wrong decisions whose consequences are catastrophic (for example, it can lead to the death of people). For instance, imagine that an AI is responsible for determining the dosage of a drug that needs to be given to a patient, based on the conditions of the patient. What if the AI makes a wrong prediction and this leads to the death of the patient? Who will be responsible for such an action? In order to accept the dosage prediction of the AI, the doctors need to <em>trust</em> the AI, but trust only comes with understanding, which requires an explanation. So, to avoid such possible failures, it is fundamental to understand the inner workings of the AI, so that it does not make those wrong decisions again.</p></li>
  <li><p>AI often needs to interact with humans, which are sentient beings (we have feelings) and that often need an explanation or reassurance (regarding some topic or event).</p></li>
  <li><p>In general, humans are often looking for an explanation and understanding of their surroundings and the world. By nature, we are curious and exploratory beings. Why does an apple fall?</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-31T14:54:43.283Z
Answers.LastActivityDate: 2019-08-31T15:20:16.763Z
Answers.LastEditDate: 2019-08-31T15:20:16.763Z
Answers.UserDisplayName: nbro
Answers.Body: <p>In addition to all these answers mentioning the more practical reasons of why we'd want explainable AIs, I'd like to add a more philosophical one. </p>
  
  <p>Understanding how things around us work is one of the main driving forces of science from antiquity. If you don't have an understanding of how things work, you can't evolve beyond that point. Just because "gravity works" hasn't stopped us trying to understand how it works. In turn a better understanding of it led to several key discoveries, which have helped us advance our technology.</p>
  
  <p>Likewise, if we stop at "it works" we will stop improving it.</p>
  
  <hr>
  
  <p>Edit:</p>
  
  <p>AI hasn't been just about making "machines think", but also through them to understand how the human brain works. AI and neuroscience go <a href="https://deepmind.com/blog/article/ai-and-neuroscience-virtuous-circle" rel="nofollow noreferrer">hand-by-hand</a>.</p>
  
  <p>This all wouldn't be possible without being able to explain AI.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T23:28:20.200Z
Answers.LastActivityDate: 2019-09-04T19:02:18.310Z
Answers.LastEditDate: 2019-09-04T19:02:18.310Z
Answers.UserDisplayName: Jerome
Answers.Body: <p>I have already given <a href="https://ai.stackexchange.com/a/14226/2444">an answer</a> and there are <a href="https://ai.stackexchange.com/a/14247/2444">other good answers</a>, but I would like to give another answer by quoting an excerpt from an old paper by <a href="https://en.wikipedia.org/wiki/Norbert_Wiener" rel="nofollow noreferrer">Norbert Wiener</a>, i.e. <a href="https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf" rel="nofollow noreferrer">Some Moral and Technical Consequences of Automation</a> (<a href="https://science.sciencemag.org/content/131/3410/1355" rel="nofollow noreferrer">1960, Science</a>)</p>
  <blockquote>
  <p>As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of performance may be delayed until long after the task which they have been set has been completed.</p>
  </blockquote>
  <blockquote>
  <p>This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant. To be effective in warding off disastrous consequences, our understanding of our man-made machines should in general develop <a href="https://en.wikipedia.org/wiki/Pari_passu" rel="nofollow noreferrer"><em>pari passu</em></a> with the performance of the machine. By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-29T14:40:39.247Z
Answers.LastActivityDate: 2020-10-29T14:40:39.247Z
Answers.UserDisplayName: nbro
Answers.Body: <p>IMHO, the most important need for explainable AI is to prevent us from becoming intellectually lazy.  If we stop <em>trying</em> to understand how answers are found, we have conceded the game to our machines.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T11:01:44.050Z
Answers.Text: Well, we understand how they find answers on a high level: They guess around millions of times until they find a promising pattern. The intellectual task is to understand WHAT CAUSES those patters - a question an AI coulnd't care less about.
Answers.UserDisplayName: Zsolt Szilagy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T13:42:57.793Z
Answers.Text: Sometimes that's true, but some types of AI (e.g., genetic algorithms) often do better than that and can be structured to provide good clues re what causes patterns.  Sometimes simply pointing out that "the pattern found in this data set is very closely analogous to the pattern found in this other data set" can be very informative and lead to an explanation of cause.
Answers.UserDisplayName: S. McGrew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T12:36:43.300Z
Answers.LastActivityDate: 2019-09-01T12:36:43.300Z
Answers.UserDisplayName: S. McGrew
Answers.Body: <p>As argued by <a href="https://arxiv.org/pdf/1610.02391.pdf" rel="nofollow noreferrer">Selvaraju et al.</a>, there are three stages of AI evolution, in which interpretability is helpful.</p>
  <ol>
  <li><p>In the early stages of AI development, when AI is weaker than human performance, transparency can help us <strong>build better models</strong>. It can give a better understanding of how a model works and helps us answer several key questions. For example, <em>why</em> a model works in some cases and doesn't in others, <em>why</em> some examples confuse the model more than others, <em>why</em> these types of models work and the others don't, etc.</p>
  </li>
  <li><p>When AI is on par with human performance and ML models are starting to be deployed in several industries, it can help build <strong>trust</strong> for these models. I'll elaborate a bit on this later, because I think that it is the most important reason.</p>
  </li>
  <li><p>When AI significantly outperforms humans (e.g. AI playing chess or Go), it can help with <strong>machine teaching</strong> (i.e. learning from the machine on how to improve human performance on that specific task).</p>
  </li>
  </ol>
  <h3>Why is trust so important?</h3>
  <p>First, let me give you a couple of examples of industries where <em>trust</em> is paramount:</p>
  <ul>
  <li><p>In healthcare, imagine a Deep Neural Net performing diagnosis for a specific disease. A classic <em>black box</em> NN would just output a binary &quot;yes&quot; or &quot;no&quot;. Even if it could outperform humans in sheer predictability, it would be utterly useless in practice. What if the doctor disagreed with the model's assessment, shouldn't he know <em>why</em> the model made that prediction; maybe it saw something the doctor missed. Furthermore, if it made a misdiagnosis (e.g. a sick person was classified as healthy and didn't get the proper treatment), who would take responsibility: the model's user? the hospital? the company that designed the model? The legal framework surrounding this is a bit blurry.</p>
  </li>
  <li><p>Another example is self-driving cars. The same questions arise: if a car crashes, whose fault is it: the driver's? the car manufacturer's? the company that designed the AI? Legal accountability, is key for the development of this industry.</p>
  </li>
  </ul>
  <p>In fact, according to many, this lack of trust has hindered the <strong>adoption</strong> of AI in many fields (sources: <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.8362&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">[1]</a>, <a href="https://www.researchgate.net/publication/3454453_Autonomy_and_Common_Ground_in_Human-Robot_Interaction_A_Field_Study" rel="nofollow noreferrer">[2]</a>, <a href="https://arxiv.org/abs/1706.07269" rel="nofollow noreferrer">[3]</a>). While there is a running hypothesis that with more transparent, interpretable or explainable systems users will be <strong>better equipped to understand and therefore trust</strong> the intelligent agents (sources: <a href="https://www.arl.army.mil/arlreports/2014/ARL-TR-6905.pdf" rel="nofollow noreferrer">[4]</a>, <a href="https://journals.sagepub.com/doi/10.1177/0018720815621206" rel="nofollow noreferrer">[5]</a>, <a href="http://www.bradhayes.info/papers/hri17.pdf" rel="nofollow noreferrer">[6]</a>).</p>
  <p>In several real-world applications, you can't just say &quot;it works 94% of the time&quot;. You might also need to provide a justification...</p>
  <h3>Government regulations</h3>
  <p>Several governments are slowly proceeding to regulate AI and transparency seems to be at the center of all of this.</p>
  <p>The first to move in this direction is the EU, which has set several guidelines where they state that AI should be transparent (sources: <a href="https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai" rel="nofollow noreferrer">[7]</a>, <a href="https://ec.europa.eu/commission/news/artificial-intelligence-2019-apr-08_en" rel="nofollow noreferrer">[8]</a>, <a href="http://www.europarl.europa.eu/news/en/press-room/20190214IPR26425/online-platforms-required-by-law-to-be-more-transparent-with-eu-businesses" rel="nofollow noreferrer">[9]</a>). For instance, the GDPR states that if a person's data has been subject to &quot;automated decision-making&quot; or &quot;profiling&quot; systems, then he has a right to access</p>
  <blockquote>
  <p>&quot;meaningful information about the logic involved&quot;</p>
  </blockquote>
  <p>(<a href="http://www.privacy-regulation.eu/en/article-15-right-of-access-by-the-data-subject-GDPR.htm" rel="nofollow noreferrer">Article 15, EU GDPR</a>)</p>
  <p>Now, this is a bit blurry, but there is clearly the intent of requiring some form of explainability from these systems. The general idea the EU is trying to pass is that &quot;if you have an automated decision-making system affecting people's lives then they have a right to know <em>why</em> a certain decision has been made.&quot; For example, a bank has an AI accepting and declining loan applications, then the applicants have a right to know <em>why</em> their application was rejected.</p>
  <h3>To sum up...</h3>
  <p>Explainable AIs are necessary because:</p>
  <ul>
  <li>It gives us a better understanding, which helps us improve them.</li>
  <li>In some cases, we can learn from AI how to make better decisions in some tasks.</li>
  <li>It helps users <em>trust</em> AI, which leads to a wider adoption of AI.</li>
  <li>Deployed AIs in the (not too distant) future might be required to be more &quot;transparent&quot;.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T23:19:06.223Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105228/discussion-on-answer-by-djib2011-why-do-we-need-explainable-ai).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T18:44:47.510Z
Answers.LastActivityDate: 2022-01-13T23:25:12.120Z
Answers.LastEditDate: 2022-01-13T23:25:12.120Z
Answers.UserDisplayName: Djib2011
Body: <p>If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the intelligence helps us to the best of their abilities and carefully arrives at its decisions, why would we need to know how its intelligence works?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T23:15:00.830Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105227/discussion-on-question-by-malioboro-why-do-we-need-explainable-ai).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-23T13:45:51.437Z
Comments.Text: Did you try to find papers on google scholar on this topic? There is some research in this area.
Comments.UserDisplayName: emely_pi
Tags: <philosophy><social><explainable-ai>
Title: Why do we need explainable AI?
UserDisplayName: malioboro
ThreadText: # Why do we need explainable AI?
  
  (Asked by: username_0 on 2019-08-31)
  
  If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the intelligence helps us to the best of their abilities and carefully arrives at its decisions, why would we need to know how its intelligence works?
  
  **Tags:** philosophy, social, explainable-ai
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105227/discussion-on-question-by-malioboro-why-do-we-need-explainable-ai).  
  > **username_2**: Did you try to find papers on google scholar on this topic? There is some research in this area.  
  
  ---
  
  ## 9 answers
  
  ### Answer by username_11 on 2019-09-01. Score: 75 | Accepted answer
  
  As argued by [Selvaraju et al.](https://arxiv.org/pdf/1610.02391.pdf), there are three stages of AI evolution, in which interpretability is helpful.
  
  1. In the early stages of AI development, when AI is weaker than human performance, transparency can help us **build better models**. It can give a better understanding of how a model works and helps us answer several key questions. For example, *why* a model works in some cases and doesn't in others, *why* some examples confuse the model more than others, *why* these types of models work and the others don't, etc.
  2. When AI is on par with human performance and ML models are starting to be deployed in several industries, it can help build **trust** for these models. I'll elaborate a bit on this later, because I think that it is the most important reason.
  3. When AI significantly outperforms humans (e.g. AI playing chess or Go), it can help with **machine teaching** (i.e. learning from the machine on how to improve human performance on that specific task).
  
  ### Why is trust so important?
  
  First, let me give you a couple of examples of industries where *trust* is paramount:
  
  * In healthcare, imagine a Deep Neural Net performing diagnosis for a specific disease. A classic *black box* NN would just output a binary "yes" or "no". Even if it could outperform humans in sheer predictability, it would be utterly useless in practice. What if the doctor disagreed with the model's assessment, shouldn't he know *why* the model made that prediction; maybe it saw something the doctor missed. Furthermore, if it made a misdiagnosis (e.g. a sick person was classified as healthy and didn't get the proper treatment), who would take responsibility: the model's user? the hospital? the company that designed the model? The legal framework surrounding this is a bit blurry.
  * Another example is self-driving cars. The same questions arise: if a car crashes, whose fault is it: the driver's? the car manufacturer's? the company that designed the AI? Legal accountability, is key for the development of this industry.
  
  In fact, according to many, this lack of trust has hindered the **adoption** of AI in many fields (sources: [[1]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.61.8362&rep=rep1&type=pdf), [[2]](https://www.researchgate.net/publication/3454453_Autonomy_and_Common_Ground_in_Human-Robot_Interaction_A_Field_Study), [[3]](https://arxiv.org/abs/1706.07269)). While there is a running hypothesis that with more transparent, interpretable or explainable systems users will be **better equipped to understand and therefore trust** the intelligent agents (sources: [[4]](https://www.arl.army.mil/arlreports/2014/ARL-TR-6905.pdf), [[5]](https://journals.sagepub.com/doi/10.1177/0018720815621206), [[6]](http://www.bradhayes.info/papers/hri17.pdf)).
  
  In several real-world applications, you can't just say "it works 94% of the time". You might also need to provide a justification...
  
  ### Government regulations
  
  Several governments are slowly proceeding to regulate AI and transparency seems to be at the center of all of this.
  
  The first to move in this direction is the EU, which has set several guidelines where they state that AI should be transparent (sources: [[7]](https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai), [[8]](https://ec.europa.eu/commission/news/artificial-intelligence-2019-apr-08_en), [[9]](http://www.europarl.europa.eu/news/en/press-room/20190214IPR26425/online-platforms-required-by-law-to-be-more-transparent-with-eu-businesses)). For instance, the GDPR states that if a person's data has been subject to "automated decision-making" or "profiling" systems, then he has a right to access
  
  > "meaningful information about the logic involved"
  
  ([Article 15, EU GDPR](http://www.privacy-regulation.eu/en/article-15-right-of-access-by-the-data-subject-GDPR.htm))
  
  Now, this is a bit blurry, but there is clearly the intent of requiring some form of explainability from these systems. The general idea the EU is trying to pass is that "if you have an automated decision-making system affecting people's lives then they have a right to know *why* a certain decision has been made." For example, a bank has an AI accepting and declining loan applications, then the applicants have a right to know *why* their application was rejected.
  
  ### To sum up...
  
  Explainable AIs are necessary because:
  
  * It gives us a better understanding, which helps us improve them.
  * In some cases, we can learn from AI how to make better decisions in some tasks.
  * It helps users *trust* AI, which leads to a wider adoption of AI.
  * Deployed AIs in the (not too distant) future might be required to be more "transparent".
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105228/discussion-on-answer-by-djib2011-why-do-we-need-explainable-ai).  
  
  ---
  
  ### Answer by username_3 on 2019-09-02. Score: 18
  
  > Why do we need explainable AI?
  > ... why we need to know "how does its intelligence work?"
  
  Because anyone with access to the equipment, enough skill, and enough time, can force the system to make a decision that is unexpected. The owner of the equipment, or 3rd parties, relying on the decision without an explanation as to why it is correct would be at a disadvantage.
  
  Examples - Someone ***might*** discover:
  
  * People whom are named John Smith and request heart surgery on: Tuesday mornings, Wednesday afternoons, or Fridays on odd days and months have a 90% chance of moving to the front of the line.
  * Couples whom have the male's last name an odd letter in the first half of the alphabet and apply for a loan with a spouse whose first name begins with a letter from the beginning of the alphabet are 40% more likely to receive the loan if they have fewer than 5 bad entries in their credit history.
  * etc.
  
  Notice that the above examples *ought not* to be determining factors in regards to the question being asked, yet it's possible for an adversary (with their own equipment, or knowledge of the algorithm) to exploit it.
  
  **Source papers**:
  
  * "[AdvHat: Real-world adversarial attack on ArcFace Face ID system](https://arxiv.org/abs/1908.08705)" (Aug 23 2019) by Stepan Komkov and Aleksandr Petiushko
  
    + Creating a sticker and placing it on your hat fools facial recognition system.
  * "[Defending against Adversarial Attacks through Resilient Feature Regeneration](https://arxiv.org/abs/1906.03444)" (Jun 8 2019), by Tejas Borkar, Felix Heide, and Lina Karam
  
    + > "Deep neural network (DNN) predictions have been shown to be vulnerable to carefully crafted adversarial perturbations. Specifically, so-called universal adversarial perturbations are image-agnostic perturbations that can be added to any image and can fool a target network into making erroneous predictions. Departing from existing adversarial defense strategies, which work in the image domain, we present a novel defense which operates in the DNN feature domain and effectively defends against such universal adversarial attacks. Our approach identifies pre-trained convolutional features that are most vulnerable to adversarial noise and deploys defender units which transform (regenerate) these DNN filter activations into noise-resilient features, guarding against unseen adversarial perturbations.".
  * "[One pixel attack for fooling deep neural networks](https://arxiv.org/abs/1710.08864)" (May 3 2019), by Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi
  
    + Altering **one pixel** can cause these errors:
    > [![Figure 1](https://i.sstatic.net/X9yf8.jpg)](https://i.sstatic.net/X9yf8.jpg)  
    > Fig. 1. One-pixel attacks created with the proposed algorithm that successfully fooled three types of DNNs trained on CIFAR-10 dataset: The All convolutional network (AllConv), Network in network (NiN) and VGG. The original class labels are in black color while the target class labels and the corresponding confidence are given below.
    >
    > [![Figure 2](https://i.sstatic.net/6RdaP.jpg)](https://i.sstatic.net/6RdaP.jpg)  
    > Fig. 2. One-pixel attacks on ImageNet dataset where the modified pixels are highlighted with red circles. The original class labels are in black color while the target class labels and their corresponding confidence are given below.
  
  Without an explanation as to how and why a decision is arrived at the decision can't be absolutely relied upon.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105229/discussion-on-answer-by-rob-why-do-we-need-explainable-ai).  
  
  ---
  
  ### Answer by username_4 on 2019-09-01. Score: 9
  
  If you're a bank, hospital or any other entity that uses predictive analytics to make a decision about actions that have huge impact on people's lives, you would not make important decisions just because Gradient Boosted trees told you to do so. Firstly, because it's risky and the underlying model might be wrong and, secondly, because in some cases it is illegal - see [Right to explanation](https://en.wikipedia.org/wiki/Right_to_explanation).
  
  
  ---
  
  ### Answer by username_1 on 2019-08-31. Score: 7
  
  Explainable AI is often desirable because
  
  1. AI (in particular, artificial neural networks) can [catastrophically fail](https://en.wikipedia.org/wiki/Catastrophic_failure) to do their intended job. More specifically, it can be hacked or attacked with [adversarial examples](https://arxiv.org/pdf/1312.6199.pdf) or it can take unexpected wrong decisions whose consequences are catastrophic (for example, it can lead to the death of people). For instance, imagine that an AI is responsible for determining the dosage of a drug that needs to be given to a patient, based on the conditions of the patient. What if the AI makes a wrong prediction and this leads to the death of the patient? Who will be responsible for such an action? In order to accept the dosage prediction of the AI, the doctors need to *trust* the AI, but trust only comes with understanding, which requires an explanation. So, to avoid such possible failures, it is fundamental to understand the inner workings of the AI, so that it does not make those wrong decisions again.
  2. AI often needs to interact with humans, which are sentient beings (we have feelings) and that often need an explanation or reassurance (regarding some topic or event).
  3. In general, humans are often looking for an explanation and understanding of their surroundings and the world. By nature, we are curious and exploratory beings. Why does an apple fall?
  
  
  ---
  
  ### Answer by username_5 on 2019-08-31. Score: 4
  
  Another reason: In the future, AI might be used for tasks that are not possible to be understood by human beings, by understanding how given AI algorithm works on that problem we might understand the nature of the given phenomenon.
  
  
  ---
  
  ### Answer by username_6 on 2019-09-02. Score: 4
  
  The answer to this is incredibly simple. If you are a bank executive one day you may need to stand up in court and explain why your AI denied mortgages to all these people... who just happen to share some protected characteristic under anti-discrimination legislation. The judge will not be happy if you handwave the question away mumbling something about algorithms. Or worse, why did this car/plane crash and how will you prevent it next time.
  
  This is the major blocker to more widespread adoption of AI in many industries.
  
  > **Comments:**
  > **username_7**: "The answer" is overstating your case. I would upvote if you said "One answer ..."  
  
  ---
  
  ### Answer by username_8 on 2019-09-02. Score: 2
  
  In addition to all these answers mentioning the more practical reasons of why we'd want explainable AIs, I'd like to add a more philosophical one.
  
  Understanding how things around us work is one of the main driving forces of science from antiquity. If you don't have an understanding of how things work, you can't evolve beyond that point. Just because "gravity works" hasn't stopped us trying to understand how it works. In turn a better understanding of it led to several key discoveries, which have helped us advance our technology.
  
  Likewise, if we stop at "it works" we will stop improving it.
  
  ---
  
  Edit:
  
  AI hasn't been just about making "machines think", but also through them to understand how the human brain works. AI and neuroscience go [hand-by-hand](https://deepmind.com/blog/article/ai-and-neuroscience-virtuous-circle).
  
  This all wouldn't be possible without being able to explain AI.
  
  
  ---
  
  ### Answer by username_1 on 2020-10-29. Score: 0
  
  I have already given [an answer](https://ai.stackexchange.com/a/14226/2444) and there are [other good answers](https://ai.stackexchange.com/a/14247/2444), but I would like to give another answer by quoting an excerpt from an old paper by [Norbert Wiener](https://en.wikipedia.org/wiki/Norbert_Wiener), i.e. [Some Moral and Technical Consequences of Automation](https://www.cs.umd.edu/users/gasarch/BLOGPAPERS/moral.pdf) ([1960, Science](https://science.sciencemag.org/content/131/3410/1355))
  
  > As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of performance may be delayed until long after the task which they have been set has been completed.
  
  > This means that though machines are theoretically subject to human criticism, such criticism may be ineffective until long after it is relevant. To be effective in warding off disastrous consequences, our understanding of our man-made machines should in general develop [*pari passu*](https://en.wikipedia.org/wiki/Pari_passu) with the performance of the machine. By the very slowness of our human actions, our effective control of our machines may be nullified. By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall.
  
  
  ---
  
  ### Answer by username_9 on 2019-09-01. Score: -1
  
  IMHO, the most important need for explainable AI is to prevent us from becoming intellectually lazy. If we stop *trying* to understand how answers are found, we have conceded the game to our machines.
  
  > **Comments:**
  > **username_10**: Well, we understand how they find answers on a high level: They guess around millions of times until they find a promising pattern. The intellectual task is to understand WHAT CAUSES those patters - a question an AI coulnd't care less about.  
  > **username_9**: Sometimes that's true, but some types of AI (e.g., genetic algorithms) often do better than that and can be structured to provide good clues re what causes patterns.  Sometimes simply pointing out that "the pattern found in this data set is very closely analogous to the pattern found in this other data set" can be very informative and lead to an explanation of cause.  
  
  ---
  
  