Answers.Body: <p>The order in which features are learned by <strong>DNNs</strong> during training is typically random &amp; depends on a number of factors:</p>
  <ul>
  <li>Including the specific architecture of the DNN</li>
  <li>The type of data being used for training</li>
  <li>&amp; the optimization algorithm being used</li>
  </ul>
  <blockquote>
  <p>However, it is known that <em>features are learned in a hierarchical manner</em>, with <strong>lower-level</strong>(simpler) features being learned first, followed by <strong>higher-level</strong>(complex) features.</p>
  </blockquote>
  <p>There is some evidence that certain features are learned early on in the training process, while others are learned later. For example, a study by Karpathy et al. (2014) found that a convolutional neural network (CNN) trained on the MNIST dataset tends to learn to distinguish between the digits 0 and 1 before it learns to distinguish between the digits 0 and 8. However, it is not clear how generalizable these findings are to other tasks and datasets.</p>
  <p>A good general resource on <strong>how different features are learned at different stages in neural networks</strong> is the <code>Deep Learning book by Goodfellow, Bengio, and Courville</code>. In particular, <strong>Chapter-6</strong> (Feature Representations) and <strong>Chapter-7</strong> (Neural Networks) cover this topic in detail.</p>
  <p><code>Artificial Intelligence: A Modern Approach by Russell and Norvig (2010)</code> covers this topic in <strong>Chapter 18</strong> on Learning Complexity. The chapter starts with a discussion of the concept of concept learning and the ways in which different concepts can be learned. It then covers the question of how well different concepts can be learned, and how this affects the overall learning process.</p>
  <p>One paper that explores this question is &quot;<a href="https://link.springer.com/chapter/10.1007/978-3-319-10590-1_53" rel="noreferrer">Visualizing and Understanding Convolutional Networks&quot; by Matthew D. Zeiler and Rob Fergus</a>.</p>
  <blockquote>
  <p>In this paper, the authors visualize the features learned by a convolutional neural network as it is trained on different datasets. They find that the network gradually learns increasingly complex features, starting with simple edge detectors and then moving on to more complex shapes and patterns.</p>
  </blockquote>
  <p><a href="https://arxiv.org/pdf/1608.08225.pdf" rel="noreferrer">Another paper Why does deep and cheap learning work so well? by Henry W. Lin, Max Tegmark, and David Rolnick</a></p>
  <blockquote>
  <p>In this paper, the author tries to answer the question of why deep learning works so well, despite being &quot;cheap&quot; (in terms of the amount of data and computation required). He argues that deep learning is able to learn a wide range of features, from low-level features (e.g. edge detectors) to high-level features (e.g. object detectors), and that this enables it to achieve better performance than shallower methods.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T21:09:27.710Z
Answers.Text: This is a really thorough wide-ranging response, and I really appreciate the time you spent crafting it. I've not going to accept it as the answer just yet though, because I'm particularly interested in empirical investigations of what content is learned at different points in the training process, as opposed to what concepts are represented in which layers. This could be relative to image classification, language generation, whatever, I'm just looking for papers that have attempted to quantify it in any way.
Answers.UserDisplayName: jon_simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T21:35:40.457Z
Answers.Text: Thanks for your appreciation, answer acceptance is secondary... not an issue :) just trying to help...here is another paper  "The Layer-wise Relevance Propagation Toolbox for Artificial Neural Networks(https://www.researchgate.net/publication/309201826_The_Layer-wise_Relevance_Propagation_Toolbox_for_Artificial_Neural_Networks) paper discuss about LRP method of decomposing the output of an ANN into the contributions of each input feature.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T22:05:20.660Z
Answers.Text: There are a few different ways to investigate what content is learned at different points in the training process. One approach is to look at the weights of the neurons in the different layers of the network. This can give you some information about which concepts are represented in which layers. Another approach is to look at the output of the network at different points in the training process. This can give you some information about what content is learned at different points in the training process.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T19:31:14.153Z
Answers.LastActivityDate: 2022-10-17T19:31:14.153Z
Answers.UserDisplayName: Faizy
Body: <p>I'm looking for papers probing into the question of what features get learned when (or equivalently what subproblems get &quot;solved&quot; when) during the training process. For example, a paper showing that a Convnet trained on MNIST learns to distinguish 0 from 1 before it learns to distinguish 0 from 8.</p>
  <p>The one example I can think of off the top of my head is the <a href="https://arxiv.org/abs/2201.02177" rel="nofollow noreferrer">Grokking paper</a>, but that's looking at a slightly different (and less intuitive) phenomenon. Thanks!</p>
  
Tags: <deep-learning><training><feature-extraction><representation-learning>
Title: Order of features learned by DNNs during training?
UserDisplayName: jon_simon
ThreadText: # Order of features learned by DNNs during training?
  
  (Asked by: username_0 on 2022-10-17)
  
  I'm looking for papers probing into the question of what features get learned when (or equivalently what subproblems get "solved" when) during the training process. For example, a paper showing that a Convnet trained on MNIST learns to distinguish 0 from 1 before it learns to distinguish 0 from 8.
  
  The one example I can think of off the top of my head is the [Grokking paper](https://arxiv.org/abs/2201.02177), but that's looking at a slightly different (and less intuitive) phenomenon. Thanks!
  
  **Tags:** deep-learning, training, feature-extraction, representation-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-17. Score: 3
  
  The order in which features are learned by **DNNs** during training is typically random & depends on a number of factors:
  
  * Including the specific architecture of the DNN
  * The type of data being used for training
  * & the optimization algorithm being used
  
  > However, it is known that *features are learned in a hierarchical manner*, with **lower-level**(simpler) features being learned first, followed by **higher-level**(complex) features.
  
  There is some evidence that certain features are learned early on in the training process, while others are learned later. For example, a study by Karpathy et al. (2014) found that a convolutional neural network (CNN) trained on the MNIST dataset tends to learn to distinguish between the digits 0 and 1 before it learns to distinguish between the digits 0 and 8. However, it is not clear how generalizable these findings are to other tasks and datasets.
  
  A good general resource on **how different features are learned at different stages in neural networks** is the `Deep Learning book by Goodfellow, Bengio, and Courville`. In particular, **Chapter-6** (Feature Representations) and **Chapter-7** (Neural Networks) cover this topic in detail.
  
  `Artificial Intelligence: A Modern Approach by Russell and Norvig (2010)` covers this topic in **Chapter 18** on Learning Complexity. The chapter starts with a discussion of the concept of concept learning and the ways in which different concepts can be learned. It then covers the question of how well different concepts can be learned, and how this affects the overall learning process.
  
  One paper that explores this question is "[Visualizing and Understanding Convolutional Networks" by Matthew D. Zeiler and Rob Fergus](https://link.springer.com/chapter/10.1007/978-3-319-10590-1_53).
  
  > In this paper, the authors visualize the features learned by a convolutional neural network as it is trained on different datasets. They find that the network gradually learns increasingly complex features, starting with simple edge detectors and then moving on to more complex shapes and patterns.
  
  [Another paper Why does deep and cheap learning work so well? by Henry W. Lin, Max Tegmark, and David Rolnick](https://arxiv.org/pdf/1608.08225.pdf)
  
  > In this paper, the author tries to answer the question of why deep learning works so well, despite being "cheap" (in terms of the amount of data and computation required). He argues that deep learning is able to learn a wide range of features, from low-level features (e.g. edge detectors) to high-level features (e.g. object detectors), and that this enables it to achieve better performance than shallower methods.
  
  > **Comments:**
  > **username_0**: This is a really thorough wide-ranging response, and I really appreciate the time you spent crafting it. I've not going to accept it as the answer just yet though, because I'm particularly interested in empirical investigations of what content is learned at different points in the training process, as opposed to what concepts are represented in which layers. This could be relative to image classification, language generation, whatever, I'm just looking for papers that have attempted to quantify it in any way.  
  > **username_1**: Thanks for your appreciation, answer acceptance is secondary... not an issue :) just trying to help...here is another paper  "The Layer-wise Relevance Propagation Toolbox for Artificial Neural Networks(https://www.researchgate.net/publication/309201826_The_Layer-wise_Relevance_Propagation_Toolbox_for_Artificial_Neural_Networks) paper discuss about LRP method of decomposing the output of an ANN into the contributions of each input feature.  
  > **username_1**: There are a few different ways to investigate what content is learned at different points in the training process. One approach is to look at the weights of the neurons in the different layers of the network. This can give you some information about which concepts are represented in which layers. Another approach is to look at the output of the network at different points in the training process. This can give you some information about what content is learned at different points in the training process.  
  
  ---
  
  