Solved by AlperEkmekci in post #6 
              
              
                You are searching for saliency map which is the color map image that involves most feature when the prediction is happening. 
You can refer this link for these.