Body: <p>I've been trying to learn backpropagation for CNNs. I read several articles like <a href="https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c" rel="nofollow noreferrer">this one</a> and <a href="https://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/" rel="nofollow noreferrer">this one</a>. They all say that to compute the gradients for the filters, you just do a convolution with the input volume as input and the error matrix as the kernel. After that, you just subtract the filter weights by the gradients(multiplied by the learning rate). I implemented this process but it's not working.</p>
  
  <p>Here's a simple example that I tried: </p>
  
  <p>Input volume (randomised)</p>
  
  <pre><code> 1 -1  0
   0  1  0
   0 -1  1
  </code></pre>
  
  <p>In this case, we want the filter to only pick up the top left 4 elements. So the target output will be: </p>
  
  <pre><code> 1  0(supposed to be -1, but ReLU is applied) 
   0  1
  </code></pre>
  
  <p>We know that the desired filter is: </p>
  
  <pre><code> 1  0
   0  0
  </code></pre>
  
  <p>But we pretend that we don't know this. </p>
  
  <p>We first randomise a filter: </p>
  
  <pre><code> 1 -1
   1  1
  </code></pre>
  
  <p>The output right now is: </p>
  
  <pre><code> 3  0 
  -2  1
  </code></pre>
  
  <p>Apply ReLU: </p>
  
  <pre><code> 3  0
   0  1
  </code></pre>
  
  <p>Error (target - output): </p>
  
  <pre><code>-2  0
   0  0
  </code></pre>
  
  <p>Use error as kernel to compute gradients: </p>
  
  <pre><code>-2  2
   0 -2
  </code></pre>
  
  <p>Say the learning rate is 0.5, then the new filter is: </p>
  
  <pre><code>2 -2
  1  2
  </code></pre>
  
  <p>This is still wrong! It's not improving at all. If this process is repeated, it won't learn the desired filter. So I must have understood the math wrong. So what's the problem here? </p>
  
Tags: <deep-learning><convolutional-neural-networks><backpropagation><math>
Title: How are filters weights updated for a CNN?
UserDisplayName: Yixuan Wang
ThreadText: # How are filters weights updated for a CNN?
  
  (Asked by: username_0 on 2019-05-20)
  
  I've been trying to learn backpropagation for CNNs. I read several articles like this one and this one. They all say that to compute the gradients for the filters, you just do a convolution with the input volume as input and the error matrix as the kernel. After that, you just subtract the filter weights by the gradients(multiplied by the learning rate). I implemented this process but it's not working.
  Here's a simple example that I tried: 
  Input volume (randomised)
   1 -1  0
   0  1  0
   0 -1  1
  
  In this case, we want the filter to only pick up the top left 4 elements. So the target output will be: 
   1  0(supposed to be -1, but ReLU is applied) 
   0  1
  
  We know that the desired filter is: 
   1  0
   0  0
  
  But we pretend that we don't know this. 
  We first randomise a filter: 
   1 -1
   1  1
  
  The output right now is: 
   3  0 
  -2  1
  
  Apply ReLU: 
   3  0
   0  1
  
  Error (target - output): 
  -2  0
   0  0
  
  Use error as kernel to compute gradients: 
  -2  2
   0 -2
  
  Say the learning rate is 0.5, then the new filter is: 
  2 -2
  1  2
  
  This is still wrong! It's not improving at all. If this process is repeated, it won't learn the desired filter. So I must have understood the math wrong. So what's the problem here?
  
  **Tags:** deep-learning, convolutional-neural-networks, backpropagation, math
  
  ---
  
  ## 0 answers
  
  