<p>It is assumed that NNs build up a hierarchical representation, whereby each layer combines features from the lower-level layers. The layers could be understood as representing a cascade of stacked features:</p>
<p>edges -&gt; texture -&gt; patterns -&gt; parts -&gt; objects</p>
<p>So from lower-level patterns to the more abstract higher-level concept like representation. This <a href="https://distill.pub/2017/feature-visualization/" rel="nofollow noreferrer">Distill article</a> as far as I can tell is one of the most cited sources (740 citations) and provides an in-depth explanation of the features and how to visualize them. The journal is peer-reviewed.</p>
<p>The post also points to some older references such as: <a href="https://www.researchgate.net/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network" rel="nofollow noreferrer">this</a>, <a href="https://arxiv.org/pdf/1312.6034.pdf" rel="nofollow noreferrer">this</a> or <a href="https://www.auduno.com/2015/07/29/visualizing-googlenet-classes/" rel="nofollow noreferrer">this</a>.   The <a href="https://colah.github.io/" rel="nofollow noreferrer">website of Chris Olah</a> one of the authors of the Distill article is also a great source for finding visualizations for different deep learning architectures.</p>

CC BY-SA 4.0
2022-08-22T07:40:00.647Z
2022-08-22T07:40:00.647Z
Mariusmarten
<p>Let's consider a deep convolutional network. It seems that there is some consensus on the following notions:</p>
<p><strong>1. Shallow layers tend to recognise more low-level features such as edges and curves.</strong></p>
<p><strong>2. Deeper layers tend to recognise more high-level features (whatever this means).</strong></p>
<p>While I usually come across various online articles and blogs that state this, no one ever cites literature that supports this claim. I am not seeking the question as to why this phenomenon happens, I'm only seeking whether it has actually been experimented on and documented. Also, I am barely able to find any peer-reviewed literature that provides evidence of this on sites such as Google Scholar or ResearchGate.</p>
<p>Could anyone point me to the right direction?</p>

CC BY-SA 4.0
2022-08-22T07:45:51.917Z
https://arxiv.org/abs/2106.14587 is the most exhausting, most fundamental work on this - it connects the architecture of NN with neural manifolds (manifolds of activities) and with categories of theories and languages. All is one system, layers (graphs, neural manifolds, languages-theories) are category theoretic fibrations. Just define learnability meatrics on this structurue and this can solve the problem of finding the optimal architecutre for the task.
TomR
<deep-learning><convolutional-neural-networks><reference-request><features><representation-learning>
Is there any proper literature on the types of features that different layers of a deep neural network learn?
mesllo
# Is there any proper literature on the types of features that different layers of a deep neural network learn?

(Asked by: username_0 on 2022-06-28)

Let's consider a deep convolutional network. It seems that there is some consensus on the following notions:

**1. Shallow layers tend to recognise more low-level features such as edges and curves.**

**2. Deeper layers tend to recognise more high-level features (whatever this means).**

While I usually come across various online articles and blogs that state this, no one ever cites literature that supports this claim. I am not seeking the question as to why this phenomenon happens, I'm only seeking whether it has actually been experimented on and documented. Also, I am barely able to find any peer-reviewed literature that provides evidence of this on sites such as Google Scholar or ResearchGate.

Could anyone point me to the right direction?

**Tags:** deep-learning, convolutional-neural-networks, reference-request, features, representation-learning

> **Comments:**
> **username_1**: https://arxiv.org/abs/2106.14587 is the most exhausting, most fundamental work on this - it connects the architecture of NN with neural manifolds (manifolds of activities) and with categories of theories and languages. All is one system, layers (graphs, neural manifolds, languages-theories) are category theoretic fibrations. Just define learnability meatrics on this structurue and this can solve the problem of finding the optimal architecutre for the task.  

---

## 1 answers

### Answer by username_2 on 2022-08-22. Score: 1

It is assumed that NNs build up a hierarchical representation, whereby each layer combines features from the lower-level layers. The layers could be understood as representing a cascade of stacked features:

edges -> texture -> patterns -> parts -> objects

So from lower-level patterns to the more abstract higher-level concept like representation. This [Distill article](https://distill.pub/2017/feature-visualization/) as far as I can tell is one of the most cited sources (740 citations) and provides an in-depth explanation of the features and how to visualize them. The journal is peer-reviewed.

The post also points to some older references such as: [this](https://www.researchgate.net/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network), [this](https://arxiv.org/pdf/1312.6034.pdf) or [this](https://www.auduno.com/2015/07/29/visualizing-googlenet-classes/). The [website of Chris Olah](https://colah.github.io/) one of the authors of the Distill article is also a great source for finding visualizations for different deep learning architectures.


---

