Answers.Body: <p>There are methods called "scoring systems" where you give a image scores such as "0.9 stripes, 0.0 red, 0.8 hair, ..." and use those scores to classify objects. It's an older idea, not used to determine if the network is learning. It's not in a standard CNN.</p>
  
  <p>To determine if relevant information is being learned or not, it's standard to use the testing accuracy, training accuracy, confusion matrix, or AUC.</p>
  
  <p>Determining what exactly a CNN is learning is a complicated research problem that's ongoing. In short - you can't really know. For a basic network, you can tell that it is learning <em>something</em> but not what it's actually using to make determinations. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-16T04:48:02.030Z
Answers.LastActivityDate: 2019-02-04T19:08:11.777Z
Answers.LastEditDate: 2019-02-04T19:08:11.777Z
Answers.UserDisplayName: Alina Barnett
Answers.Body: <p>Yes there is! If a model generalizes well to the test set, we already know that it has found <em>some</em> useful features. However, the latent representation of the data may still be "entangled" - a single element of the latent vector may actually encode information about multiple attributes of the input, or a single attribute may be spread across multiple elements.  We usually prefer a representation in which the features are represented by the axes of the latent space - a "disentangled" representation.  For example, if we were encoding faces, it would be nice to have an axis for smiling/not, another for masculine/feminine, and ao on.</p>
  
  <p>Pushing models to learn "clean" (disentangled) representations is an active sub-field of machine learning research with practical applications (like interpretability, but also because it makes it easier for "downstream" models to learn their tasks, e.g. a control policy in reinforcement learning system taking as input a learned representation from a <a href="https://worldmodels.github.io/" rel="nofollow noreferrer">world model</a>).  </p>
  
  <p>Where to begin?  Start with L2 regularisation to push your network to "spend" those weights wisely (more weights close to zero => sparser latent vector) and work your way up from there.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-05T09:56:53.143Z
Answers.LastActivityDate: 2019-02-05T09:56:53.143Z
Answers.UserDisplayName: Edward Dixon
Answers.Body: <p>So there's pictures of low level activation maps, and some gradient based information where yoy take the deriviative of the output with respect to the input and generate a heatmap.</p>
  
  <p>I kind of have my doubts on how usefull this is in general, imo it kind of is creating a fallacious illusion of understanding.</p>
  
  <p>There's some additional research using blurring to figure out the relevant features also but again I have my doubts.</p>
  
  <p>Probably the most usefull is generating images by optimizing your class score. You can learn how badly your CNN actually labels things(doing this makes you realize quickly that CNN are garbage at actually understanding and incredibly easy to trick).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-19T04:05:27.760Z
Answers.LastActivityDate: 2020-05-19T04:05:27.760Z
Answers.UserDisplayName: FourierFlux
Body: <p>Is there any way to control the extraction of features? How do I determine which features are been learned during training, i.e relevant information is been learned or not?</p>
  
Tags: <deep-learning><convolutional-neural-networks><feature-extraction>
Title: How do I determine which relevant features have been learned during training in a CNN?
UserDisplayName: Rohit Jere
ThreadText: # How do I determine which relevant features have been learned during training in a CNN?
  
  (Asked by: username_0 on 2019-01-15)
  
  Is there any way to control the extraction of features? How do I determine which features are been learned during training, i.e relevant information is been learned or not?
  
  **Tags:** deep-learning, convolutional-neural-networks, feature-extraction
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-01-16. Score: 3
  
  There are methods called "scoring systems" where you give a image scores such as "0.9 stripes, 0.0 red, 0.8 hair, ..." and use those scores to classify objects. It's an older idea, not used to determine if the network is learning. It's not in a standard CNN.
  
  To determine if relevant information is being learned or not, it's standard to use the testing accuracy, training accuracy, confusion matrix, or AUC.
  
  Determining what exactly a CNN is learning is a complicated research problem that's ongoing. In short - you can't really know. For a basic network, you can tell that it is learning *something* but not what it's actually using to make determinations.
  
  
  ---
  
  ### Answer by username_2 on 2019-02-05. Score: 0
  
  Yes there is! If a model generalizes well to the test set, we already know that it has found *some* useful features. However, the latent representation of the data may still be "entangled" - a single element of the latent vector may actually encode information about multiple attributes of the input, or a single attribute may be spread across multiple elements. We usually prefer a representation in which the features are represented by the axes of the latent space - a "disentangled" representation. For example, if we were encoding faces, it would be nice to have an axis for smiling/not, another for masculine/feminine, and ao on.
  
  Pushing models to learn "clean" (disentangled) representations is an active sub-field of machine learning research with practical applications (like interpretability, but also because it makes it easier for "downstream" models to learn their tasks, e.g. a control policy in reinforcement learning system taking as input a learned representation from a [world model](https://worldmodels.github.io/)).
  
  Where to begin? Start with L2 regularisation to push your network to "spend" those weights wisely (more weights close to zero => sparser latent vector) and work your way up from there.
  
  
  ---
  
  ### Answer by username_3 on 2020-05-19. Score: 0
  
  So there's pictures of low level activation maps, and some gradient based information where yoy take the deriviative of the output with respect to the input and generate a heatmap.
  
  I kind of have my doubts on how usefull this is in general, imo it kind of is creating a fallacious illusion of understanding.
  
  There's some additional research using blurring to figure out the relevant features also but again I have my doubts.
  
  Probably the most usefull is generating images by optimizing your class score. You can learn how badly your CNN actually labels things(doing this makes you realize quickly that CNN are garbage at actually understanding and incredibly easy to trick).
  
  
  ---
  
  