Answers.Body: <p>The reason is that a layer can only learn a function based on it's inputs - that's clear. When you look at a convolutional neural network, each output pixel of an early layer only gets a very small fraction of the whole image as input. For example: If your first layer uses a <code>3 x 3</code> kernel, then each output only depends on only 9 pixels. Try detecting anything other then edges with that much information - it's not not possible. In other words: Looking at a 3 by 3 image you will be able to see an edge and its orientation, but not something like a nose.</p>
  <p>In contrast, the later layers depend on a much larger portion of the image and can thus learn high-level features like a nose.</p>
  <p>It's more a question of whats possible for the layers to learn and not a question of the network self-organizing it's resources.
  And besides just reasoning like that you can visualize what the kernels in the individual layers respond to. I found <a href="https://ai.stackexchange.com/questions/28272/are-these-visualisations-the-filters-of-the-convolution-layer-or-the-convolved-i">this question</a> that also has an image with some further information on that.</p>
  <p><strong>EDIT</strong> <em>In response to the first comment:</em></p>
  <p>To further clarify how a CNN works, here is an example, where <span class="math-container">$X_0$</span> is the input image and <span class="math-container">$X_1$</span> is the output of the first CNN-Layer:</p>
  <ol>
  <li>Each convolution in the first layer receives a small (e.g. <code>3 x 3</code>) area of pixels of <span class="math-container">$X_0$</span> and provides an output pixel. Here it is only possible to detect simple features (edges / gradients) because of the reason stated above.</li>
  <li>The output of the first Layer <span class="math-container">$X_1$</span> is again an image, with the following properties:
  <ul>
  <li>A single pixel represents a <code>3 x 3</code> area in the original input</li>
  <li>Effectively <span class="math-container">$X_1$</span> is a map that encodes the location of the simple features that got detected.</li>
  </ul>
  </li>
  <li>Because of the above properties, applying another <code>3 x 3</code> convolution to <span class="math-container">$X_1$</span> causes that
  <ul>
  <li>That convolution effectively depends on a <code>5 x 5</code> area of <span class="math-container">$X_0$</span>, i.e. a <code>3 x 3</code> area of <span class="math-container">$X_1$</span></li>
  <li>That convolution responds to certain arrangements of these 1st-level features, e.g. two adjacent edges with different orientations are a corner. You can think of the CNN-layers as a hierarchy where initial layers provide basic features the next layer detects compositions of these, the next layer detects compositions of the compositions and so on.</li>
  </ul>
  </li>
  </ol>
  <blockquote>
  <p>is it like the many basic functions (like the edge detections ) are added together and then create a function that fires high values when a &quot;nose&quot; detected ?</p>
  </blockquote>
  <p>So yes, you can say it like this. This is exactly where the thought of hierarchy fits in.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-08T13:37:07.993Z
Answers.Text: So what you are saying is as layers progress the function learned become more and more complex in terms of number of input pixels , which can be regarded as capturing more complex features such as nose right ? The only thing that I did not understand how the coarser features are related to the high level features in the network , is it like the many  basic functions (like the edge detections ) are added together and then create a function that fires high values when a "nose" detected ?
Answers.UserDisplayName: levitatmas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-08T14:09:49.400Z
Answers.Text: Yes, you can say that - I extended the answer to clarify this a bit more
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-08T12:03:33.027Z
Answers.LastActivityDate: 2022-07-08T14:09:19.230Z
Answers.LastEditDate: 2022-07-08T14:09:19.230Z
Answers.UserDisplayName: Chillston
Body: <p>I was reading about convolutional neural networks and I came across such an explanation:</p>
  <blockquote>
  <p>Consider detecting features in human face. The earlier layers of neural networks learn coarse features such as edges in the images and the latent layers learn more complex ( finer) features such as eyes, nose and etc</p>
  </blockquote>
  <p>I am wondering why this is a true statement, namely how can we know that a neural network first starts by learning primitive features and then learns complex features. Could you please explain?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-10T17:32:41.023Z
Comments.Text: Where did you find that explanation? Can you provide the source?
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><convolutional-neural-networks><representation-learning>
Title: How do neural networks learn specific features throughout the layers?
UserDisplayName: levitatmas
ThreadText: # How do neural networks learn specific features throughout the layers?
  
  (Asked by: username_0 on 2022-07-08)
  
  I was reading about convolutional neural networks and I came across such an explanation:
  
  > Consider detecting features in human face. The earlier layers of neural networks learn coarse features such as edges in the images and the latent layers learn more complex ( finer) features such as eyes, nose and etc
  
  I am wondering why this is a true statement, namely how can we know that a neural network first starts by learning primitive features and then learns complex features. Could you please explain?
  
  **Tags:** neural-networks, machine-learning, convolutional-neural-networks, representation-learning
  > **Comments:**
  > **username_1**: Where did you find that explanation? Can you provide the source?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-07-08. Score: 2 | Accepted answer
  
  The reason is that a layer can only learn a function based on it's inputs - that's clear. When you look at a convolutional neural network, each output pixel of an early layer only gets a very small fraction of the whole image as input. For example: If your first layer uses a `3 x 3` kernel, then each output only depends on only 9 pixels. Try detecting anything other then edges with that much information - it's not not possible. In other words: Looking at a 3 by 3 image you will be able to see an edge and its orientation, but not something like a nose.
  
  In contrast, the later layers depend on a much larger portion of the image and can thus learn high-level features like a nose.
  
  It's more a question of whats possible for the layers to learn and not a question of the network self-organizing it's resources.
  And besides just reasoning like that you can visualize what the kernels in the individual layers respond to. I found [this question](https://ai.stackexchange.com/questions/28272/are-these-visualisations-the-filters-of-the-convolution-layer-or-the-convolved-i) that also has an image with some further information on that.
  
  **EDIT** *In response to the first comment:*
  
  To further clarify how a CNN works, here is an example, where $X\_0$ is the input image and $X\_1$ is the output of the first CNN-Layer:
  
  1. Each convolution in the first layer receives a small (e.g. `3 x 3`) area of pixels of $X\_0$ and provides an output pixel. Here it is only possible to detect simple features (edges / gradients) because of the reason stated above.
  2. The output of the first Layer $X\_1$ is again an image, with the following properties:
     * A single pixel represents a `3 x 3` area in the original input
     * Effectively $X\_1$ is a map that encodes the location of the simple features that got detected.
  3. Because of the above properties, applying another `3 x 3` convolution to $X\_1$ causes that
     * That convolution effectively depends on a `5 x 5` area of $X\_0$, i.e. a `3 x 3` area of $X\_1$
     * That convolution responds to certain arrangements of these 1st-level features, e.g. two adjacent edges with different orientations are a corner. You can think of the CNN-layers as a hierarchy where initial layers provide basic features the next layer detects compositions of these, the next layer detects compositions of the compositions and so on.
  
  > is it like the many basic functions (like the edge detections ) are added together and then create a function that fires high values when a "nose" detected ?
  
  So yes, you can say it like this. This is exactly where the thought of hierarchy fits in.
  
  > **Comments:**
  > **username_0**: So what you are saying is as layers progress the function learned become more and more complex in terms of number of input pixels , which can be regarded as capturing more complex features such as nose right ? The only thing that I did not understand how the coarser features are related to the high level features in the network , is it like the many  basic functions (like the edge detections ) are added together and then create a function that fires high values when a "nose" detected ?  
  > **username_2**: Yes, you can say that - I extended the answer to clarify this a bit more  
  
  ---
  
  