Answers.Body: <p>There are broadly two phenomena that are often referred to as not understanding deep learning:</p>
  <ol>
  <li><p>It is not well understood how the network arrives at any particular solution. Trying to analyze the weights into human understandable knowledge is often futile. The model acts as a black box: it seems to do well, but we do not understand the rules it uses to operate. The field of AI that attempts to deal with this is called <a href="https://en.m.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="nofollow noreferrer">explainability</a>.</p>
  </li>
  <li><p>It is not well understood why neural networks, which are heavily over parametrized, do not seem to overfit. The field of study that attempts to tackle this is called <a href="https://en.m.wikipedia.org/wiki/Computational_learning_theory" rel="nofollow noreferrer">Computational Learning Theory</a>.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T23:52:04.663Z
Answers.LastActivityDate: 2023-03-13T23:52:04.663Z
Answers.UserDisplayName: chessprogrammer
Body: <p>It is a sentence that I hear a lot and I guess I don't get what it means.</p>
  <p>It seems that the weight optimization procedure is very well understood and there is, to some extent, theoretical/empirical foundations that justify the different techniques.</p>
  <p>So what exactly about DNNs is not understood?</p>
  <p>Any pointing to a reference would already be helpful.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-13T22:09:04.243Z
Comments.Text: What's definitely not understood: in which part of the matrix multiplications does sentient thought occur?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-15T16:24:20.973Z
Comments.Text: Usually this is referring to Explainability.
Comments.UserDisplayName: Rexcirus
Tags: <deep-learning><philosophy><agi>
Title: What is meant by "Deep Learning models are not understood"?
UserDisplayName: benr
ThreadText: # What is meant by "Deep Learning models are not understood"?
  
  (Asked by: username_0 on 2023-03-13)
  
  It is a sentence that I hear a lot and I guess I don't get what it means.
  
  It seems that the weight optimization procedure is very well understood and there is, to some extent, theoretical/empirical foundations that justify the different techniques.
  
  So what exactly about DNNs is not understood?
  
  Any pointing to a reference would already be helpful.
  
  **Tags:** deep-learning, philosophy, agi
  > **Comments:**
  > **username_1**: What's definitely not understood: in which part of the matrix multiplications does sentient thought occur?  
  > **username_2**: Usually this is referring to Explainability.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-03-13. Score: 3 | Accepted answer
  
  There are broadly two phenomena that are often referred to as not understanding deep learning:
  
  1. It is not well understood how the network arrives at any particular solution. Trying to analyze the weights into human understandable knowledge is often futile. The model acts as a black box: it seems to do well, but we do not understand the rules it uses to operate. The field of AI that attempts to deal with this is called [explainability](https://en.m.wikipedia.org/wiki/Explainable_artificial_intelligence).
  2. It is not well understood why neural networks, which are heavily over parametrized, do not seem to overfit. The field of study that attempts to tackle this is called [Computational Learning Theory](https://en.m.wikipedia.org/wiki/Computational_learning_theory).
  
  
  ---
  
  