Answers.Body: <p>I think you are misreading the relevant passage here.</p>
  <p>Since you do not specify exact excerpt(s), I take that by &quot;<em>implicit assumption</em>&quot; you refer to the equation (2) (application of a ReLU) and the corresponding text explanation (bold emphasis mine):</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/NZXHl.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NZXHl.jpg" alt="enter image description here" /></a></p>
  <p>We apply a ReLU to the linear combination of maps because we are only
  interested in the features that have a <em>positive</em> influence on the class
  of interest, i.e. pixels whose intensity should be <em>increased</em> in order
  to increase <span class="math-container">$y^c$</span>. <strong>Negative pixels are likely to belong to other
  categories in the image</strong>. As expected, without this ReLU, localization maps sometimes highlight more than just the desired class and perform worse at localization.</p>
  </blockquote>
  <p>The first thing to notice here is that this choice is not at all about activations close to zero, as you seem to believe,  but about <strong>negative</strong> ones; and since <em>negative</em> activations are indeed likely to belong to other categories/classes than the one being &quot;explained&quot; at a given trial, it is very natural to exclude them using a ReLU.</p>
  <p>Grad-CAM maps are essentially <em>localization</em> ones; this is apparent already from the paper abstract (emphasis mine):</p>
  <blockquote>
  <p>Our approach – Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’  in  a classification  network  or  a  sequence  of  words in captioning network) flowing into the final convolutional layer to produce a coarse <strong>localization</strong> map highlighting the important regions in the image for predicting the concept.</p>
  </blockquote>
  <p>and they are even occasionally referred to as &quot;Grad-CAM <em>localizations</em>&quot; (e.g. in the caption of Fig. 14); taking a standard example figure from the paper, e.g. this part of Fig. 1:</p>
  <p><a href="https://i.sstatic.net/ldZeR.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ldZeR.jpg" alt="enter image description here" /></a></p>
  <p>it is hard to see how including the negative values of the map (i.e. removing the <em>thresholding</em> imposed by the ReLU) would not lead to maps that include irrelevant parts of the image, hence resulting in a worse localization.</p>
  <hr />
  <p>A general remark: while your claim that</p>
  <blockquote>
  <p>After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means</p>
  </blockquote>
  <p>is correct as long as we treat the network as an arbitrary mathematical model, we can no longer treat a <em>trained</em> network as such. For a trained network (which Grad-CAM is all about), the exact values of both biases &amp; weights matter, and we cannot transform them arbitrarily.</p>
  <hr />
  <p><strong>UPDATE</strong> (after comments):</p>
  <blockquote>
  <p>Are you pointing out that it's called &quot;localization&quot; and therefore must be so?</p>
  </blockquote>
  <p>It is called &quot;localization&quot; because it <em>is</em> localization, literally (&quot;look, <em>here</em> is the &quot;dog&quot; in the picture, <em>not</em> there&quot;).</p>
  <blockquote>
  <p>I could make a similar challenge &quot;why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?&quot;</p>
  </blockquote>
  <p>It is not at all like that; positive means X and negative means not-X <em>in the presence of class X</em> (i.e. a specific X is present), in the <em>specific</em> network, and all this in a localization context; notice that Grad-CAM for &quot;dog&quot; is different from the one for &quot;cat&quot; in the picture above.</p>
  <blockquote>
  <p>Why is it that a trained network tends to make 0 mean &quot;insignficant&quot; in the deeper layers? [...] why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?</p>
  </blockquote>
  <p>Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are <em>broken</em>; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific <strong>one-hot encoding</strong> of the labels: we have encoded &quot;cat&quot; and &quot;dog&quot; as (say) <code>[0, 1]</code> and <code>[1, 0]</code> respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the <em>positive</em> activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as <code>[0, -1]</code> and <code>[-1, 0]</code> respectively (&quot;minus-one-hot encoding&quot;), then yes, your argument would hold, and we would be interested in the <em>negative</em> activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side (and thus broken the symmetry)...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-29T15:40:42.337Z
Answers.Text: Good, thank you. A few challenges to help me understand: 1) "this choice is not at all about activations close to zero, as you seem to believe". Fine (my mistake), but I could make a similar challenge "why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?" 2) Am I missing something about your "localization" argument? It seems like semantics to me. Are you pointing out that it's called "localization" and therefore must be so? 3) On your general remark. That's the crux of my question I think ...
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-29T15:44:23.727Z
Answers.Text: ... ie. Why is it that a trained network tends to make 0 mean "insignficant" in the deeper layers? And to go back on challenge (1), why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?
  Let me know if my challenges make sense, or if I failed to understand your points.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-29T17:28:22.830Z
Answers.Text: Got you. Although now I remember why I was focusing so much on 0, and I never thought about negative and positive. I drew a truth table for this and you can try it too (I'll post it in OP). If $\alpha_c^k < 0$ AND $A^{k}_{i,j} < 0$ then you get a +ve in the product. Grad-CAM doesn't care what the activations alone are. It cares about the product of the activations and gradients. Now I disagree with the paper's statement too "Negative pixels are likely to belong to other categories in the image." Maybe by "pixel" they mean after taking the product.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-29T17:37:14.360Z
Answers.Text: @AlexanderSoare I think they mean exactly that (they use the argument to justify the ReLU).
Answers.UserDisplayName: desertnaut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-29T17:40:44.153Z
Answers.Text: Cool, then... I guess I'm happy now. Kind of reminds me Paulo Coelho's The Alchemist.
Answers.UserDisplayName: Alexander Soare
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-29T15:25:04.033Z
Answers.LastActivityDate: 2020-11-29T17:08:28.297Z
Answers.LastEditDate: 2020-11-29T17:08:28.297Z
Answers.UserDisplayName: desertnaut
Body: <p>In the <a href="https://arxiv.org/pdf/1610.02391.pdf" rel="nofollow noreferrer">GradCAM paper</a> section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification problem). That is:</p>
  <ul>
  <li><span class="math-container">$A^k$</span> the final feature maps</li>
  <li><span class="math-container">$\alpha_k^c$</span> the average pooled partial derivatives of the output class scores <span class="math-container">$y^c$</span> with respect to the the final feature maps <span class="math-container">$A_k$</span>.</li>
  </ul>
  <p>The second point is clear to me. The stronger the derivative, the more important the <span class="math-container">$k$</span>th channel of the final feature maps is.</p>
  <p>The first point is not, because the implicit assumption is that non-zero activations have more significance than activations close to zero. I know it's tempting to take that as a given, but for me it's not so obvious. After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means. We can easily transform two neurons [0, 1] to [1, 0] with a linear transformation.</p>
  <p>So why should it matter which regions of the final feature maps are strongly activated?</p>
  <hr />
  <p><strong>EDIT</strong></p>
  <p>To address a comment further down, this table explains why I'm thinking about <strong>magnitude</strong> rather than <strong>sign</strong> of the activations.</p>
  <p><a href="https://i.sstatic.net/ZjQB7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZjQB7.png" alt="enter image description here" /></a></p>
  <p>It comes from thinking about the possible variations of</p>
  <p><span class="math-container">$$
  L_{Grad-CAM}^c = ReLU\bigl( \sum_k \alpha_k^c A^k \bigr)
  $$</span></p>
  
Tags: <neural-networks><deep-neural-networks><explainable-ai><grad-cam>
Title: In GradCAM, why is activation strength considered an indicator of relevant regions?
UserDisplayName: Alexander Soare
ThreadText: # In GradCAM, why is activation strength considered an indicator of relevant regions?
  
  (Asked by: username_0 on 2020-11-20)
  
  In the [GradCAM paper](https://arxiv.org/pdf/1610.02391.pdf) section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification problem). That is:
  
  * $A^k$ the final feature maps
  * $\alpha\_k^c$ the average pooled partial derivatives of the output class scores $y^c$ with respect to the the final feature maps $A\_k$.
  
  The second point is clear to me. The stronger the derivative, the more important the $k$th channel of the final feature maps is.
  
  The first point is not, because the implicit assumption is that non-zero activations have more significance than activations close to zero. I know it's tempting to take that as a given, but for me it's not so obvious. After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means. We can easily transform two neurons [0, 1] to [1, 0] with a linear transformation.
  
  So why should it matter which regions of the final feature maps are strongly activated?
  
  ---
  
  **EDIT**
  
  To address a comment further down, this table explains why I'm thinking about **magnitude** rather than **sign** of the activations.
  
  [![enter image description here](https://i.sstatic.net/ZjQB7.png)](https://i.sstatic.net/ZjQB7.png)
  
  It comes from thinking about the possible variations of
  
  $$
  L\_{Grad-CAM}^c = ReLU\bigl( \sum\_k \alpha\_k^c A^k \bigr)
  $$
  
  **Tags:** neural-networks, deep-neural-networks, explainable-ai, grad-cam
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-29. Score: 1 | Accepted answer
  
  I think you are misreading the relevant passage here.
  
  Since you do not specify exact excerpt(s), I take that by "*implicit assumption*" you refer to the equation (2) (application of a ReLU) and the corresponding text explanation (bold emphasis mine):
  
  > [![enter image description here](https://i.sstatic.net/NZXHl.jpg)](https://i.sstatic.net/NZXHl.jpg)
  >
  > We apply a ReLU to the linear combination of maps because we are only
  > interested in the features that have a *positive* influence on the class
  > of interest, i.e. pixels whose intensity should be *increased* in order
  > to increase $y^c$. **Negative pixels are likely to belong to other
  > categories in the image**. As expected, without this ReLU, localization maps sometimes highlight more than just the desired class and perform worse at localization.
  
  The first thing to notice here is that this choice is not at all about activations close to zero, as you seem to believe, but about **negative** ones; and since *negative* activations are indeed likely to belong to other categories/classes than the one being "explained" at a given trial, it is very natural to exclude them using a ReLU.
  
  Grad-CAM maps are essentially *localization* ones; this is apparent already from the paper abstract (emphasis mine):
  
  > Our approach – Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’ in a classification network or a sequence of words in captioning network) flowing into the final convolutional layer to produce a coarse **localization** map highlighting the important regions in the image for predicting the concept.
  
  and they are even occasionally referred to as "Grad-CAM *localizations*" (e.g. in the caption of Fig. 14); taking a standard example figure from the paper, e.g. this part of Fig. 1:
  
  [![enter image description here](https://i.sstatic.net/ldZeR.jpg)](https://i.sstatic.net/ldZeR.jpg)
  
  it is hard to see how including the negative values of the map (i.e. removing the *thresholding* imposed by the ReLU) would not lead to maps that include irrelevant parts of the image, hence resulting in a worse localization.
  
  ---
  
  A general remark: while your claim that
  
  > After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means
  
  is correct as long as we treat the network as an arbitrary mathematical model, we can no longer treat a *trained* network as such. For a trained network (which Grad-CAM is all about), the exact values of both biases & weights matter, and we cannot transform them arbitrarily.
  
  ---
  
  **UPDATE** (after comments):
  
  > Are you pointing out that it's called "localization" and therefore must be so?
  
  It is called "localization" because it *is* localization, literally ("look, *here* is the "dog" in the picture, *not* there").
  
  > I could make a similar challenge "why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?"
  
  It is not at all like that; positive means X and negative means not-X *in the presence of class X* (i.e. a specific X is present), in the *specific* network, and all this in a localization context; notice that Grad-CAM for "dog" is different from the one for "cat" in the picture above.
  
  > Why is it that a trained network tends to make 0 mean "insignficant" in the deeper layers? [...] why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?
  
  Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are *broken*; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific **one-hot encoding** of the labels: we have encoded "cat" and "dog" as (say) `[0, 1]` and `[1, 0]` respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the *positive* activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as `[0, -1]` and `[-1, 0]` respectively ("minus-one-hot encoding"), then yes, your argument would hold, and we would be interested in the *negative* activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side (and thus broken the symmetry)...
  
  > **Comments:**
  > **username_0**: Good, thank you. A few challenges to help me understand: 1) "this choice is not at all about activations close to zero, as you seem to believe". Fine (my mistake), but I could make a similar challenge "why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?" 2) Am I missing something about your "localization" argument? It seems like semantics to me. Are you pointing out that it's called "localization" and therefore must be so? 3) On your general remark. That's the crux of my question I think ...  
  > **username_0**: ... ie. Why is it that a trained network tends to make 0 mean "insignficant" in the deeper layers? And to go back on challenge (1), why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)? Let me know if my challenges make sense, or if I failed to understand your points.  
  > **username_0**: Got you. Although now I remember why I was focusing so much on 0, and I never thought about negative and positive. I drew a truth table for this and you can try it too (I'll post it in OP). If $\alpha_c^k < 0$ AND $A^{k}_{i,j} < 0$ then you get a +ve in the product. Grad-CAM doesn't care what the activations alone are. It cares about the product of the activations and gradients. Now I disagree with the paper's statement too "Negative pixels are likely to belong to other categories in the image." Maybe by "pixel" they mean after taking the product.  
  > **username_1**: @AlexanderSoare I think they mean exactly that (they use the argument to justify the ReLU).  
  > **username_0**: Cool, then... I guess I'm happy now. Kind of reminds me Paulo Coelho's The Alchemist.  
  
  ---
  
  