Answers.Body: <p>I ended up using Guided backprop from TorchRay.
  <a href="https://facebookresearch.github.io/TorchRay/attribution.html#module-torchray.attribution.guided_backprop" rel="nofollow noreferrer">https://facebookresearch.github.io/TorchRay/attribution.html#module-torchray.attribution.guided_backprop</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-29T01:15:50.117Z
Answers.Text: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-28T23:29:53.713Z
Answers.LastActivityDate: 2022-06-28T23:29:53.713Z
Answers.UserDisplayName: samuel fipps
Body: <p>I have a couple opportunities to write a paper, or papers over some of the neural networks I have made.</p>
  <p>I was wondering if there are anyways to figure out why the neural network classifies the data I have the way it does. As in, what features of the data the neural network is using to classify the data. The neural networks I'm using consists of ltsm layers, CNN layers, and fc layers.</p>
  <p>I have thought about plotting the neural network at everything output, but this doesn't really help to much. Just because there are so many weights that go to each node in a layer it makes it very hard to determine what's happening. I could plot the bias's but I don't know how much influence they have over the weights.</p>
  <p>Another thing I considered was adjusting the the values of the input data a little bit at a time and seeing where the classification changes. Which this would work to some extent, but wouldn't allow to me to get the full picture of what the neural network is doing.</p>
  <p>So any suggestions on how to do this?</p>
  <p>Or any papers that would be help over this topic!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-27T20:58:18.273Z
Comments.Text: Or any papers that would be help over this topic
Comments.UserDisplayName: samuel fipps
Tags: <neural-networks><convolutional-neural-networks><long-short-term-memory><deep-neural-networks>
Title: How to identify important features in data?
UserDisplayName: samuel fipps
ThreadText: # How to identify important features in data?
  
  (Asked by: username_0 on 2022-06-27)
  
  I have a couple opportunities to write a paper, or papers over some of the neural networks I have made.
  
  I was wondering if there are anyways to figure out why the neural network classifies the data I have the way it does. As in, what features of the data the neural network is using to classify the data. The neural networks I'm using consists of ltsm layers, CNN layers, and fc layers.
  
  I have thought about plotting the neural network at everything output, but this doesn't really help to much. Just because there are so many weights that go to each node in a layer it makes it very hard to determine what's happening. I could plot the bias's but I don't know how much influence they have over the weights.
  
  Another thing I considered was adjusting the the values of the input data a little bit at a time and seeing where the classification changes. Which this would work to some extent, but wouldn't allow to me to get the full picture of what the neural network is doing.
  
  So any suggestions on how to do this?
  
  Or any papers that would be help over this topic!
  
  **Tags:** neural-networks, convolutional-neural-networks, long-short-term-memory, deep-neural-networks
  > **Comments:**
  > **username_0**: Or any papers that would be help over this topic  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-28. Score: 0 | Accepted answer
  
  I ended up using Guided backprop from TorchRay.
  <https://facebookresearch.github.io/TorchRay/attribution.html#module-torchray.attribution.guided_backprop>
  
  > **Comments:**
  > **username_1**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  