<p>There are broadly two phenomena that are often referred to as not understanding deep learning:</p>
<ol>
<li><p>It is not well understood how the network arrives at any particular solution. Trying to analyze the weights into human understandable knowledge is often futile. The model acts as a black box: it seems to do well, but we do not understand the rules it uses to operate. The field of AI that attempts to deal with this is called <a href="https://en.m.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="nofollow noreferrer">explainability</a>.</p>
</li>
<li><p>It is not well understood why neural networks, which are heavily over parametrized, do not seem to overfit. The field of study that attempts to tackle this is called <a href="https://en.m.wikipedia.org/wiki/Computational_learning_theory" rel="nofollow noreferrer">Computational Learning Theory</a>.</p>
</li>
</ol>

CC BY-SA 4.0
2023-03-13T23:52:04.663Z
2023-03-13T23:52:04.663Z
chessprogrammer
<p>It is a sentence that I hear a lot and I guess I don't get what it means.</p>
<p>It seems that the weight optimization procedure is very well understood and there is, to some extent, theoretical/empirical foundations that justify the different techniques.</p>
<p>So what exactly about DNNs is not understood?</p>
<p>Any pointing to a reference would already be helpful.</p>

CC BY-SA 4.0
2023-03-13T22:09:04.243Z
What's definitely not understood: in which part of the matrix multiplications does sentient thought occur?
Criticize SE actions means ban
CC BY-SA 4.0
2023-03-15T16:24:20.973Z
Usually this is referring to Explainability.
Rexcirus
<deep-learning><philosophy><agi>
What is meant by "Deep Learning models are not understood"?
benr
# What is meant by "Deep Learning models are not understood"?

(Asked by: username_0 on 2023-03-13)

It is a sentence that I hear a lot and I guess I don't get what it means.

It seems that the weight optimization procedure is very well understood and there is, to some extent, theoretical/empirical foundations that justify the different techniques.

So what exactly about DNNs is not understood?

Any pointing to a reference would already be helpful.

**Tags:** deep-learning, philosophy, agi

> **Comments:**
> **username_1**: What's definitely not understood: in which part of the matrix multiplications does sentient thought occur?  
> **username_2**: Usually this is referring to Explainability.  

---

## 1 answers

### Answer by username_3 on 2023-03-13. Score: 3 | Accepted answer

There are broadly two phenomena that are often referred to as not understanding deep learning:

1. It is not well understood how the network arrives at any particular solution. Trying to analyze the weights into human understandable knowledge is often futile. The model acts as a black box: it seems to do well, but we do not understand the rules it uses to operate. The field of AI that attempts to deal with this is called [explainability](https://en.m.wikipedia.org/wiki/Explainable_artificial_intelligence).
2. It is not well understood why neural networks, which are heavily over parametrized, do not seem to overfit. The field of study that attempts to tackle this is called [Computational Learning Theory](https://en.m.wikipedia.org/wiki/Computational_learning_theory).


---

