Answers.Body: <p>While this may not be the answer you were looking for, I hope this explanation will help you to understand applying backpropagation to a CNN. Fundamentally, convolutional layers are no different than dense layers, however there are restrictions. The key one is weight-sharing which allows a CNN to be much more efficient than a regular dense layer (as well as it being sparse due to locality). Imagine we are transforming a 4x4 image into a 2x2 image. Since we are inputting a 16-vector, and outputting a 4-vector, we need a weights matrix of 4x16:</p>
  
  <p><a href="https://i.sstatic.net/YHJad.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YHJad.png" alt="enter image description here"></a></p>
  
  <p>This has 64 parameters. In a convolutional layer, we can accomplish this by convolving a 3x3 kernel over the image:</p>
  
  <p><span class="math-container">$$ K=
  \begin{bmatrix}
  k_{1,1} &amp; k_{1,2} &amp; k_{1,3} \\
  k_{2,1} &amp; k_{2,2} &amp; k_{2,3} \\
  k_{3,1} &amp; k_{3,2} &amp; k_{3,3}
  \end{bmatrix} $$</span></p>
  
  <p>This convolution is equivalent to multiplying by the weights matrix:</p>
  
  <p><a href="https://i.sstatic.net/jyY0V.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jyY0V.png" alt="enter image description here"></a></p>
  
  <p>As you can see, this only requires 9 parameters and backpropagation can be applied to update these parameters.</p>
  
  <p>Image Source: <a href="https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1" rel="nofollow noreferrer">https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T08:15:45.740Z
Answers.Text: thanks but as you said it is not my question's answer.
  I finally find $dL/dX$ for each channel in my example ``16`` then I add all of them and 
  I get my $dL/dX$
Answers.UserDisplayName: A.Najafi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-22T12:20:48.813Z
Answers.LastActivityDate: 2020-05-22T12:20:48.813Z
Answers.UserDisplayName: Archie Shahidullah
Body: <p>I am trying to implement CNN using python NumPy. I searched so much, but all I found was for one filter with one channel for convolution.</p>
  <p>Suppose <span class="math-container">$x$</span> is an image with the shape: <code>(N_Height, N_Width, N_Channel) = (5,5,3)</code>.</p>
  <p>Let's say I have <code>16</code> filters with this shape: <code>(F_Height, F_Width, N_Channel) = (3,3,3)</code> , <code>stride=1</code> and <code>padding=0</code></p>
  <p><strong>Forward</strong>:</p>
  <p>The output shape after convolution 2d will be</p>
  <pre><code>(
  math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
  math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),
  filter_count
  )
  </code></pre>
  <p>So, the output of this layer will be an array with this shape: <code>(Height, Width, Channel) =  (3, 3, 16)</code></p>
  <p><strong>BackPropagation</strong>:</p>
  <p>Suppose <span class="math-container">$dL/dh$</span> is the input for my layer in back-propagation with this shape: <code>(3, 3, 16)</code></p>
  <p>Now, I must find <span class="math-container">$dL/dw$</span> and <span class="math-container">$dL/dx$</span>: <span class="math-container">$dL/dw$</span> to update my filters parameter and <span class="math-container">$dL/dx$</span> to pass it as input to the previous layer as the loss respect to the input <span class="math-container">$x$</span>.</p>
  <p>From this answer <a href="https://datascience.stackexchange.com/a/45978/89170">Error respect to filters weights</a> I found how to calculate <span class="math-container">$dL/dw$</span>.</p>
  <p>The problem I have in the back-propagation is I don't know how to calculate <span class="math-container">$dL/dx$</span> having this shape: <code>(5, 5, 3)</code> and pass it to the previous layer.</p>
  <p>I read lots of articles in Medium and other sites, but I don't get how to calculate it:</p>
  <ul>
  <li><a href="https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c" rel="nofollow noreferrer">How Backpropagation works in a CNN</a></li>
  <li><a href="https://medium.com/technologymadeeasy/the-best-explanation-of-convolutional-neural-networks-on-the-internet-fbb8b1ad5df8" rel="nofollow noreferrer">The best explanation of Convolutional Neural Networks on the Internet!</a></li>
  <li><a href="https://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/" rel="nofollow noreferrer">Backpropagation In Convolutional Neural Networks</a></li>
  <li><a href="https://datascience.stackexchange.com/questions/28593/how-to-propagate-error-back-to-previous-layer-in-cnn?rq=1">How to propagate error back to previous layer in CNN?</a></li>
  </ul>
  
Tags: <deep-learning><convolutional-neural-networks><backpropagation>
Title: How do I calculate the partial derivative with respect to $x$?
UserDisplayName: A.Najafi
ThreadText: # How do I calculate the partial derivative with respect to $x$?
  
  (Asked by: username_0 on 2020-05-16)
  
  I am trying to implement CNN using python NumPy. I searched so much, but all I found was for one filter with one channel for convolution.
  
  Suppose $x$ is an image with the shape: `(N_Height, N_Width, N_Channel) = (5,5,3)`.
  
  Let's say I have `16` filters with this shape: `(F_Height, F_Width, N_Channel) = (3,3,3)` , `stride=1` and `padding=0`
  
  **Forward**:
  
  The output shape after convolution 2d will be
  
  ```
  (
  math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
  math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),
  filter_count
  )
  
  ```
  
  So, the output of this layer will be an array with this shape: `(Height, Width, Channel) = (3, 3, 16)`
  
  **BackPropagation**:
  
  Suppose $dL/dh$ is the input for my layer in back-propagation with this shape: `(3, 3, 16)`
  
  Now, I must find $dL/dw$ and $dL/dx$: $dL/dw$ to update my filters parameter and $dL/dx$ to pass it as input to the previous layer as the loss respect to the input $x$.
  
  From this answer [Error respect to filters weights](https://datascience.stackexchange.com/a/45978/89170) I found how to calculate $dL/dw$.
  
  The problem I have in the back-propagation is I don't know how to calculate $dL/dx$ having this shape: `(5, 5, 3)` and pass it to the previous layer.
  
  I read lots of articles in Medium and other sites, but I don't get how to calculate it:
  
  * [How Backpropagation works in a CNN](https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c)
  * [The best explanation of Convolutional Neural Networks on the Internet!](https://medium.com/technologymadeeasy/the-best-explanation-of-convolutional-neural-networks-on-the-internet-fbb8b1ad5df8)
  * [Backpropagation In Convolutional Neural Networks](https://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/)
  * [How to propagate error back to previous layer in CNN?](https://datascience.stackexchange.com/questions/28593/how-to-propagate-error-back-to-previous-layer-in-cnn?rq=1)
  
  **Tags:** deep-learning, convolutional-neural-networks, backpropagation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-22. Score: 1
  
  While this may not be the answer you were looking for, I hope this explanation will help you to understand applying backpropagation to a CNN. Fundamentally, convolutional layers are no different than dense layers, however there are restrictions. The key one is weight-sharing which allows a CNN to be much more efficient than a regular dense layer (as well as it being sparse due to locality). Imagine we are transforming a 4x4 image into a 2x2 image. Since we are inputting a 16-vector, and outputting a 4-vector, we need a weights matrix of 4x16:
  
  [![enter image description here](https://i.sstatic.net/YHJad.png)](https://i.sstatic.net/YHJad.png)
  
  This has 64 parameters. In a convolutional layer, we can accomplish this by convolving a 3x3 kernel over the image:
  
  $$ K=
  \begin{bmatrix}
  k\_{1,1} & k\_{1,2} & k\_{1,3} \\
  k\_{2,1} & k\_{2,2} & k\_{2,3} \\
  k\_{3,1} & k\_{3,2} & k\_{3,3}
  \end{bmatrix} $$
  
  This convolution is equivalent to multiplying by the weights matrix:
  
  [![enter image description here](https://i.sstatic.net/jyY0V.png)](https://i.sstatic.net/jyY0V.png)
  
  As you can see, this only requires 9 parameters and backpropagation can be applied to update these parameters.
  
  Image Source: <https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1>
  
  > **Comments:**
  > **username_0**: thanks but as you said it is not my question's answer. I finally find $dL/dX$ for each channel in my example ``16`` then I add all of them and  I get my $dL/dX$  
  
  ---
  
  