Answers.Body: <p>I really liked the question. Yes, we sum over derivatives. First of all think what backpropagation is trying to do: <em>finding the affect of each parameter on the loss</em>.</p>
  <p>So as you said:</p>
  <blockquote>
  <p>the same filter is used multiple times on the input while convolving</p>
  </blockquote>
  <p>meaning that each kernel affects the final loss in several ways, so those affects should be summed together, not averaged.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T12:40:30.267Z
Answers.Text: Can you provide a link to a reliable implementation where the derivatives are summed? Moreover, I don't think this answer really clarifies why we need to sum, because you say "meaning that each kernel affects the final loss in several ways". Well, "several ways" does not imply that you need to sum the gradients. In any case, it's not necessarily true that each filter is applied to the image multiple times. It's usually applied once. What you actually do is to apply **multiple different** filters to the image, but they are usually different. So, can you clarify how you interpreted the question?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T22:08:28.660Z
Answers.Text: I don't know any reliable link to the implementation, but I've coded the backprop of conv layer myself (with numpy) and got the same result by doing so in *tensorflow* (using tf.gradients).
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T22:15:42.020Z
Answers.Text: When you apply a kernel to its input, you multiply the same kernel in different parts of input to produce the output of different neurons. So the same kernel affects the final loss in several ways (via different neurons). I think [this](https://miro.medium.com/max/2340/1*Fw-ehcNBR9byHtho-Rxbtw.gif) gif helps to understand what I want to say.
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T22:22:03.737Z
Answers.Text: Ha, yes, you apply the kernel to different parts of the image: that's just the definition of the convolution/cross-correlation. Ok, now I understand what you mean when the OP said "apply the filter multiple times".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-04T11:28:15.473Z
Answers.LastActivityDate: 2020-12-04T11:28:15.473Z
Answers.UserDisplayName: amin
Body: <p>In a Convolutional Neural Network, unlike the fully connected layers, the same filter is used multiple times on the input while convolving - so during backpropagation, we get multiple derivatives for the filter parameters w.r.t the loss function. My question is, why do we sum all the derivatives to get the final gradient? Because, we don't sum the output of the convolution during forward pass. So, isn't it more sensible to average them? What is the intuition behind this?</p>
  <p>PS: although I said CNN, what I'm actually doing is correlation for simplicity of learning.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-07T11:20:18.337Z
Comments.Text: To provide more context, can you provide the link to an implementation (or an explanation) where this is done (or stated)? Typically, you apply each filter only once to the image. You may have different filters that you apply to the image, but that's different than applying the exact same filter to the image.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T16:05:15.310Z
Comments.Text: X-posted: https://math.stackexchange.com/q/3897423
Comments.UserDisplayName: Rodrigo de Azevedo
Tags: <convolutional-neural-networks><backpropagation><convolution><filters>
Title: In CNNs, why do we sum the filter derivatives w.r.t the loss function to get the final gradient?
UserDisplayName: Manu Soman
ThreadText: # In CNNs, why do we sum the filter derivatives w.r.t the loss function to get the final gradient?
  
  (Asked by: username_0 on 2020-11-07)
  
  In a Convolutional Neural Network, unlike the fully connected layers, the same filter is used multiple times on the input while convolving - so during backpropagation, we get multiple derivatives for the filter parameters w.r.t the loss function. My question is, why do we sum all the derivatives to get the final gradient? Because, we don't sum the output of the convolution during forward pass. So, isn't it more sensible to average them? What is the intuition behind this?
  
  PS: although I said CNN, what I'm actually doing is correlation for simplicity of learning.
  
  **Tags:** convolutional-neural-networks, backpropagation, convolution, filters
  > **Comments:**
  > **username_1**: To provide more context, can you provide the link to an implementation (or an explanation) where this is done (or stated)? Typically, you apply each filter only once to the image. You may have different filters that you apply to the image, but that's different than applying the exact same filter to the image.  
  > **username_2**: X-posted: https://math.stackexchange.com/q/3897423  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-12-04. Score: 0 | Accepted answer
  
  I really liked the question. Yes, we sum over derivatives. First of all think what backpropagation is trying to do: *finding the affect of each parameter on the loss*.
  
  So as you said:
  
  > the same filter is used multiple times on the input while convolving
  
  meaning that each kernel affects the final loss in several ways, so those affects should be summed together, not averaged.
  
  > **Comments:**
  > **username_1**: Can you provide a link to a reliable implementation where the derivatives are summed? Moreover, I don't think this answer really clarifies why we need to sum, because you say "meaning that each kernel affects the final loss in several ways". Well, "several ways" does not imply that you need to sum the gradients. In any case, it's not necessarily true that each filter is applied to the image multiple times. It's usually applied once. What you actually do is to apply **multiple different** filters to the image, but they are usually different. So, can you clarify how you interpreted the question?  
  > **username_3**: I don't know any reliable link to the implementation, but I've coded the backprop of conv layer myself (with numpy) and got the same result by doing so in *tensorflow* (using tf.gradients).  
  > **username_3**: When you apply a kernel to its input, you multiply the same kernel in different parts of input to produce the output of different neurons. So the same kernel affects the final loss in several ways (via different neurons). I think [this](https://miro.medium.com/max/2340/1*Fw-ehcNBR9byHtho-Rxbtw.gif) gif helps to understand what I want to say.  
  > **username_1**: Ha, yes, you apply the kernel to different parts of the image: that's just the definition of the convolution/cross-correlation. Ok, now I understand what you mean when the OP said "apply the filter multiple times".  
  
  ---
  
  