<p>I've been trying to learn backpropagation for CNNs. I read several articles like <a href="https://medium.com/@pavisj/convolutions-and-backpropagations-46026a8f5d2c" rel="nofollow noreferrer">this one</a> and <a href="https://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/" rel="nofollow noreferrer">this one</a>. They all say that to compute the gradients for the filters, you just do a convolution with the input volume as input and the error matrix as the kernel. After that, you just subtract the filter weights by the gradients(multiplied by the learning rate). I implemented this process but it's not working.</p>

<p>Here's a simple example that I tried: </p>

<p>Input volume (randomised)</p>

<pre><code> 1 -1  0
 0  1  0
 0 -1  1
</code></pre>

<p>In this case, we want the filter to only pick up the top left 4 elements. So the target output will be: </p>

<pre><code> 1  0(supposed to be -1, but ReLU is applied) 
 0  1
</code></pre>

<p>We know that the desired filter is: </p>

<pre><code> 1  0
 0  0
</code></pre>

<p>But we pretend that we don't know this. </p>

<p>We first randomise a filter: </p>

<pre><code> 1 -1
 1  1
</code></pre>

<p>The output right now is: </p>

<pre><code> 3  0 
-2  1
</code></pre>

<p>Apply ReLU: </p>

<pre><code> 3  0
 0  1
</code></pre>

<p>Error (target - output): </p>

<pre><code>-2  0
 0  0
</code></pre>

<p>Use error as kernel to compute gradients: </p>

<pre><code>-2  2
 0 -2
</code></pre>

<p>Say the learning rate is 0.5, then the new filter is: </p>

<pre><code>2 -2
1  2
</code></pre>

<p>This is still wrong! It's not improving at all. If this process is repeated, it won't learn the desired filter. So I must have understood the math wrong. So what's the problem here? </p>

<deep-learning><convolutional-neural-networks><backpropagation><math>
How are filters weights updated for a CNN?
Yixuan Wang
# How are filters weights updated for a CNN?

(Asked by: username_0 on 2019-05-20)

I've been trying to learn backpropagation for CNNs. I read several articles like this one and this one. They all say that to compute the gradients for the filters, you just do a convolution with the input volume as input and the error matrix as the kernel. After that, you just subtract the filter weights by the gradients(multiplied by the learning rate). I implemented this process but it's not working.
Here's a simple example that I tried: 
Input volume (randomised)
 1 -1  0
 0  1  0
 0 -1  1

In this case, we want the filter to only pick up the top left 4 elements. So the target output will be: 
 1  0(supposed to be -1, but ReLU is applied) 
 0  1

We know that the desired filter is: 
 1  0
 0  0

But we pretend that we don't know this. 
We first randomise a filter: 
 1 -1
 1  1

The output right now is: 
 3  0 
-2  1

Apply ReLU: 
 3  0
 0  1

Error (target - output): 
-2  0
 0  0

Use error as kernel to compute gradients: 
-2  2
 0 -2

Say the learning rate is 0.5, then the new filter is: 
2 -2
1  2

This is still wrong! It's not improving at all. If this process is repeated, it won't learn the desired filter. So I must have understood the math wrong. So what's the problem here?

**Tags:** deep-learning, convolutional-neural-networks, backpropagation, math


---

## 0 answers

