Answers.Body: <p>It has much more to do with the <strong>inference of new data points</strong> than the training of deep neural networks(DNNs). Because DNNs learn complicated non-linear relations between your input variables and your targets, it is hard (or altogether impossible) to tell why a certain new data point has been classified as it was.</p>
  <p>Take for example a model that takes in a lot of data about your health status and tells the insurance company what your insurance premium should be. The company would like to be able to tell you that because of some factor X, your insurance premium is Y. If the model is some simple linear regression, they can tell you that e.g. because you smoke, your premium increase by XY dollars, since you can interpret the coefficient of the dummy variable: Smoker: <em>yes/no</em>.</p>
  <p>However, with a DNN model, all they can tell you is that &quot;Our model, that's very complicated and precise, told us that your premium is Y?&quot; In other words, because of a large number of parameters and non-linear activation functions you cannot tell what's the precise effect of smoking on your insurance premium. I hope this makes it clearer :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-06T13:53:55.160Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-06T12:41:48.790Z
Answers.LastActivityDate: 2022-06-06T12:41:48.790Z
Answers.UserDisplayName: ImotVoksim
Body: <p>I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I tried to do some digging myself by starting with the gradient descent algorithm as an optimization technique used to model error of predictions. Ideally, I am aware it is most suitable for predicted errors that fit a Gaussian distribution. One question I have not been able to find answers to and would really appreciate help from this community</p>
  <p>What exactly does the AI explainability problem refer to? Is it related to the convex optimisation (G.D.) process ? Or randomly generated coefficients? How exactly do we mean when we say AI is a blackbox?</p>
  <p>p.s: If this has already been answered elsewhere on this platform do share a link.</p>
  <p>Further clarification on my question:</p>
  <p>Please, reference <a href="https://twitter.com/ylecun/status/1533451405167669249" rel="nofollow noreferrer">this tweet</a> by Y. Lecun only yesterday, I would like some more responses on relationship between the AI explainability problem and the weight optimization technique. How or why is G.D applied to optimising parameter weights contribute to AI being a blackbox. Or if this is another separate problem from AI explainability also do clarify.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-06T13:51:22.847Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <neural-networks><terminology><gradient-descent><explainable-ai><black-box>
Title: What exactly is the AI explainability problem?
UserDisplayName: rp2001
ThreadText: # What exactly is the AI explainability problem?
  
  (Asked by: username_0 on 2022-06-06)
  
  I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I tried to do some digging myself by starting with the gradient descent algorithm as an optimization technique used to model error of predictions. Ideally, I am aware it is most suitable for predicted errors that fit a Gaussian distribution. One question I have not been able to find answers to and would really appreciate help from this community
  
  What exactly does the AI explainability problem refer to? Is it related to the convex optimisation (G.D.) process ? Or randomly generated coefficients? How exactly do we mean when we say AI is a blackbox?
  
  p.s: If this has already been answered elsewhere on this platform do share a link.
  
  Further clarification on my question:
  
  Please, reference [this tweet](https://twitter.com/ylecun/status/1533451405167669249) by Y. Lecun only yesterday, I would like some more responses on relationship between the AI explainability problem and the weight optimization technique. How or why is G.D applied to optimising parameter weights contribute to AI being a blackbox. Or if this is another separate problem from AI explainability also do clarify.
  
  **Tags:** neural-networks, terminology, gradient-descent, explainable-ai, black-box
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-06. Score: 1
  
  It has much more to do with the **inference of new data points** than the training of deep neural networks(DNNs). Because DNNs learn complicated non-linear relations between your input variables and your targets, it is hard (or altogether impossible) to tell why a certain new data point has been classified as it was.
  
  Take for example a model that takes in a lot of data about your health status and tells the insurance company what your insurance premium should be. The company would like to be able to tell you that because of some factor X, your insurance premium is Y. If the model is some simple linear regression, they can tell you that e.g. because you smoke, your premium increase by XY dollars, since you can interpret the coefficient of the dummy variable: Smoker: *yes/no*.
  
  However, with a DNN model, all they can tell you is that "Our model, that's very complicated and precise, told us that your premium is Y?" In other words, because of a large number of parameters and non-linear activation functions you cannot tell what's the precise effect of smoking on your insurance premium. I hope this makes it clearer :)
  
  > **Comments:**
  > **username_1**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  