Answers.Body: <p>There are many frameworks which allow you to do that.</p>
  
  <p>One of them, which supports many different techniques for visualization, can be found here: 
  <a href="https://github.com/marcoancona/DeepExplain" rel="nofollow noreferrer">https://github.com/marcoancona/DeepExplain</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-24T18:41:24.813Z
Answers.LastActivityDate: 2020-02-24T18:41:24.813Z
Answers.UserDisplayName: Vishal
Answers.Body: <p>Yes there definitely is, and research into this has actually resulted in some really cool behaviour.</p>
  
  <p>One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic segmentation.</p>
  
  <p>Other ways are to change segments of the input and to see how that affects the output (like add 0.1 to a pixel).</p>
  
  <p>You can also determine what each filter in a convolutional layer is looking at using similar techniques.</p>
  
  <p>It's a super interesting field of machine learning, and I highly recommend taking a look at <a href="https://www.youtube.com/watch?v=6wcs6szJWMY&amp;list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&amp;index=13&amp;t=0s" rel="nofollow noreferrer">this</a> lecture that is completely free and one of the most interesting ones I've personally seen. it will explain all this much better than I have done.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T02:06:17.697Z
Answers.LastActivityDate: 2020-02-20T02:06:17.697Z
Answers.UserDisplayName: Recessive
Body: <p>Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value.</p>
  
  <p><a href="https://i.sstatic.net/Z35w2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Z35w2.png" alt="enter image description here"></a></p>
  
  <p>I suppose a good model would predict a 6, but maybe with less than 100% certainty (or maybe with 100% certainty - I don't know that it matters for the purpose of this question).</p>
  
  <p>Is there any way to get information about which pixels most strongly influenced the decision of the CNN, and which pixels were not so important? So to represent that visually, green means that those pixels were important:</p>
  
  <p><a href="https://i.sstatic.net/T7XhO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/T7XhO.png" alt="enter image description here"></a></p>
  
  <p>Or conversely, is it possible to highlight pixels which did not contribute to the outcome (or which cast doubt on the outcome thereby reducing the certainty from 100%)</p>
  
  <p><a href="https://i.sstatic.net/3YpG7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3YpG7.png" alt="enter image description here"></a></p>
  
Tags: <convolutional-neural-networks><explainable-ai>
Title: Can a NN be configured to indicate which points of the input influenced its prediction and how?
UserDisplayName: Alexander Soare
ThreadText: # Can a NN be configured to indicate which points of the input influenced its prediction and how?
  
  (Asked by: username_0 on 2020-02-19)
  
  Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value.
  
  [![enter image description here](https://i.sstatic.net/Z35w2.png)](https://i.sstatic.net/Z35w2.png)
  
  I suppose a good model would predict a 6, but maybe with less than 100% certainty (or maybe with 100% certainty - I don't know that it matters for the purpose of this question).
  
  Is there any way to get information about which pixels most strongly influenced the decision of the CNN, and which pixels were not so important? So to represent that visually, green means that those pixels were important:
  
  [![enter image description here](https://i.sstatic.net/T7XhO.png)](https://i.sstatic.net/T7XhO.png)
  
  Or conversely, is it possible to highlight pixels which did not contribute to the outcome (or which cast doubt on the outcome thereby reducing the certainty from 100%)
  
  [![enter image description here](https://i.sstatic.net/3YpG7.png)](https://i.sstatic.net/3YpG7.png)
  
  **Tags:** convolutional-neural-networks, explainable-ai
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-24. Score: 1
  
  There are many frameworks which allow you to do that.
  
  One of them, which supports many different techniques for visualization, can be found here:
  <https://github.com/marcoancona/DeepExplain>
  
  
  ---
  
  ### Answer by username_2 on 2020-02-20. Score: 1
  
  Yes there definitely is, and research into this has actually resulted in some really cool behaviour.
  
  One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic segmentation.
  
  Other ways are to change segments of the input and to see how that affects the output (like add 0.1 to a pixel).
  
  You can also determine what each filter in a convolutional layer is looking at using similar techniques.
  
  It's a super interesting field of machine learning, and I highly recommend taking a look at [this](https://www.youtube.com/watch?v=6wcs6szJWMY&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=13&t=0s) lecture that is completely free and one of the most interesting ones I've personally seen. it will explain all this much better than I have done.
  
  
  ---
  
  