Answers.Body: <p><a href="https://i.sstatic.net/X9Pih.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/X9Pih.gif" alt="Animation of backpropagation with stride &gt; 1"></a></p>
  
  <p>Backpropagation with stride > 1 involves dilation of the gradient tensor with stride-1 zeroes. I created a <a href="https://medium.com/@mayank.utexas/backpropagation-for-convolution-with-strides-8137e4fc2710" rel="nofollow noreferrer">blog post</a> that describes this in greater detail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-07T21:06:39.653Z
Answers.LastActivityDate: 2019-05-07T21:06:39.653Z
Answers.UserDisplayName: Mayank
Answers.Body: <p>From the paper found from the post linked below: </p>
  
  <blockquote>
    <p>'We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks'</p>
  </blockquote>
  
  <p>All that means that only values are skipped (=pooling is made) to the matrix, otherwise all works like a convolution should do.</p>
  
  <p><strong>Sources:</strong></p>
  
  <p><a href="https://arxiv.org/pdf/1412.6806.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1412.6806.pdf</a></p>
  
  <p><a href="https://stackoverflow.com/questions/44666390/max-pool-layer-vs-convolution-with-stride-performance">https://stackoverflow.com/questions/44666390/max-pool-layer-vs-convolution-with-stride-performance</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-02T14:52:43.397Z
Answers.Text: This is not an answer to my question; you are talking about pooling, my question is about simple convolution in backpropagation algorithm.
Answers.UserDisplayName: volperossa
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-02T14:55:32.817Z
Answers.Text: Well, the point is that strides introduce pooling kind of phenomenom and otherwise it does not change CNN performance and if I read my source right, also correctness.
Answers.UserDisplayName: mico
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-02T15:01:48.587Z
Answers.Text: Probably more direct professional expertice you could get on datascience.SE site. I belong to both of these, thus I knew sth of the issue.
Answers.UserDisplayName: mico
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-02T14:47:59.657Z
Answers.LastActivityDate: 2018-04-02T14:47:59.657Z
Answers.UserDisplayName: mico
Answers.Body: <p>I have just the same problem, and I was trying to derive the backpropagation for the convolutional layer with stride, but it doesn't work.</p>
  <p>When you do the striding in the forward propagation, you chose the elements next to each other to convolve with the kernel, then take a step <span class="math-container">$&gt;1$</span>. This results in the fact that in the backpropagation, in the reverse operation, the delta matrix elements will be multiplied by the kernel elements, (with the rotation) but not as strided, but you are picking elements that are not next to each other, something like <span class="math-container">$DY_{11} * K_{11} + DY_{13} * K_{12} + DY_{31} * K_{21} + DY_{33} * K_{22}$</span>, which is NOT the equivalent as a convolution with a stride <span class="math-container">$&gt;1$</span>.</p>
  <p>So as far as I am concerned, if I would like to implement the ConvNet by myself to get a better grasp of the concept, I have to implement a different method for the backprop, if I allow strides.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-11T03:20:04.523Z
Answers.LastActivityDate: 2021-12-30T13:38:15.393Z
Answers.LastEditDate: 2021-12-30T13:38:15.393Z
Answers.UserDisplayName: Nice Micro
Body: <p>I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated by <span class="math-container">$180°$</span>, i.e. something like</p>
  <p><span class="math-container">$$\delta^l_{ij}=\delta^{l+1}_{ij} * rot180(W^{l+1})f'(x^l_{ij})$$</span></p>
  <p>with <span class="math-container">$*$</span> convolution operator. This is valid with <span class="math-container">$stride=1$</span>.</p>
  <p>However, what happens when stride is greater than <span class="math-container">$1$</span>? Is it still a convolution with a kernel rotation or I can't make this simplification?</p>
  
Tags: <convolutional-neural-networks><backpropagation><gradient-descent><convolutional-layers>
Title: How to compute the derivative of the error with respect to the input of a convolutional layer when the stride is bigger than 1?
UserDisplayName: volperossa
ThreadText: # How to compute the derivative of the error with respect to the input of a convolutional layer when the stride is bigger than 1?
  
  (Asked by: username_0 on 2018-03-22)
  
  I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated by $180°$, i.e. something like
  
  $$\delta^l\_{ij}=\delta^{l+1}\_{ij} \* rot180(W^{l+1})f'(x^l\_{ij})$$
  
  with $\*$ convolution operator. This is valid with $stride=1$.
  
  However, what happens when stride is greater than $1$? Is it still a convolution with a kernel rotation or I can't make this simplification?
  
  **Tags:** convolutional-neural-networks, backpropagation, gradient-descent, convolutional-layers
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-05-07. Score: 3
  
  [![Animation of backpropagation with stride > 1](https://i.sstatic.net/X9Pih.gif)](https://i.sstatic.net/X9Pih.gif)
  
  Backpropagation with stride > 1 involves dilation of the gradient tensor with stride-1 zeroes. I created a [blog post](https://medium.com/@mayank.utexas/backpropagation-for-convolution-with-strides-8137e4fc2710) that describes this in greater detail.
  
  
  ---
  
  ### Answer by username_2 on 2018-04-02. Score: 1
  
  From the paper found from the post linked below:
  
  > 'We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks'
  
  All that means that only values are skipped (=pooling is made) to the matrix, otherwise all works like a convolution should do.
  
  **Sources:**
  
  <https://arxiv.org/pdf/1412.6806.pdf>
  
  <https://stackoverflow.com/questions/44666390/max-pool-layer-vs-convolution-with-stride-performance>
  
  > **Comments:**
  > **username_0**: This is not an answer to my question; you are talking about pooling, my question is about simple convolution in backpropagation algorithm.  
  > **username_2**: Well, the point is that strides introduce pooling kind of phenomenom and otherwise it does not change CNN performance and if I read my source right, also correctness.  
  > **username_2**: Probably more direct professional expertice you could get on datascience.SE site. I belong to both of these, thus I knew sth of the issue.  
  
  ---
  
  ### Answer by username_3 on 2019-01-11. Score: 1
  
  I have just the same problem, and I was trying to derive the backpropagation for the convolutional layer with stride, but it doesn't work.
  
  When you do the striding in the forward propagation, you chose the elements next to each other to convolve with the kernel, then take a step $>1$. This results in the fact that in the backpropagation, in the reverse operation, the delta matrix elements will be multiplied by the kernel elements, (with the rotation) but not as strided, but you are picking elements that are not next to each other, something like $DY\_{11} \* K\_{11} + DY\_{13} \* K\_{12} + DY\_{31} \* K\_{21} + DY\_{33} \* K\_{22}$, which is NOT the equivalent as a convolution with a stride $>1$.
  
  So as far as I am concerned, if I would like to implement the ConvNet by myself to get a better grasp of the concept, I have to implement a different method for the backprop, if I allow strides.
  
  
  ---
  
  