Answers.Body: <p>All weight matrices in a Neural Network, adapt to map input and output. ReLU, as you pointed out, doesn't give negative derivatives. You're right. But notice the weight update equation in Backpropagation, it uses a multitude of parameters like:</p>
  
  <ul>
  <li>Error value</li>
  <li>Activation from the previous layer</li>
  <li>Weight matrix of the previous layer</li>
  <li>The pre-activation cache of the previous layer</li>
  </ul>
  
  <p>These can be positive or negative. Hence weight updations dW and db can be +ve or -ve, allowing for decreasing (or increasing) value of weights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-28T13:13:01.497Z
Answers.LastActivityDate: 2019-04-28T13:13:01.497Z
Answers.UserDisplayName: Karan Shah
Answers.Body: <p>The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to <span class="math-container">$-\infty$</span> and you want to approximate a function different than the one the CNN represents initially with all weights set to <span class="math-container">$-\infty$</span>. You will have to increase one or more weights. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-23T20:14:46.117Z
Answers.LastActivityDate: 2019-04-23T20:14:46.117Z
Answers.UserDisplayName: nbro
Body: <p>I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation.
  But one thing I noticed was the "just positive" derivative of the ReLU function.</p>
  
  <p>My network structure is the following:</p>
  
  <ul>
  <li>(Input 28x28)</li>
  <li>Conv Layer (filter count = 6, filter size = 3x3,  stride = 1)</li>
  <li>Max Pool Layer (Size 2x2) with RELU</li>
  <li>Conv Layer (filter count = 6, filter size = 3x3,  stride = 1)</li>
  <li>Max Pool Layer (Size 2x2) with RELU</li>
  <li>Dense (128)</li>
  <li>Dense (10)</li>
  </ul>
  
  <p>I noticed, when looking at the gradients, that the ReLU derivative is always (as it should be) positive. But is it right that the filter weights are always decreasing their weights? Or is there any way they can increase their weight?</p>
  
  <p>Whenever I look at any of the filter's values, they decreased after training. Is that correct?</p>
  
  <p>By the way, I am using stochastic gradient descent with a fixed learning rate for training.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-22T12:13:41.797Z
Comments.Text: By negative or positive direction do you mean that they respectively decrease or increase in value?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-23T01:34:10.423Z
Comments.Text: Yes that is what I meant. I'll update my post.
Comments.UserDisplayName: jutu OOtv
Tags: <convolutional-neural-networks><backpropagation><relu><convolution>
Title: How should the values of the filters of a CNN change?
UserDisplayName: jutu OOtv
ThreadText: # How should the values of the filters of a CNN change?
  
  (Asked by: username_0 on 2019-04-22)
  
  I wrote a convolutional neural network for the MNIST dataset with Numpy from scratch. I am currently trying to understand every part and calculation.
  But one thing I noticed was the "just positive" derivative of the ReLU function.
  
  My network structure is the following:
  
  * (Input 28x28)
  * Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
  * Max Pool Layer (Size 2x2) with RELU
  * Conv Layer (filter count = 6, filter size = 3x3, stride = 1)
  * Max Pool Layer (Size 2x2) with RELU
  * Dense (128)
  * Dense (10)
  
  I noticed, when looking at the gradients, that the ReLU derivative is always (as it should be) positive. But is it right that the filter weights are always decreasing their weights? Or is there any way they can increase their weight?
  
  Whenever I look at any of the filter's values, they decreased after training. Is that correct?
  
  By the way, I am using stochastic gradient descent with a fixed learning rate for training.
  
  **Tags:** convolutional-neural-networks, backpropagation, relu, convolution
  > **Comments:**
  > **username_1**: By negative or positive direction do you mean that they respectively decrease or increase in value?  
  > **username_0**: Yes that is what I meant. I'll update my post.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-04-28. Score: 0 | Accepted answer
  
  All weight matrices in a Neural Network, adapt to map input and output. ReLU, as you pointed out, doesn't give negative derivatives. You're right. But notice the weight update equation in Backpropagation, it uses a multitude of parameters like:
  
  * Error value
  * Activation from the previous layer
  * Weight matrix of the previous layer
  * The pre-activation cache of the previous layer
  
  These can be positive or negative. Hence weight updations dW and db can be +ve or -ve, allowing for decreasing (or increasing) value of weights.
  
  
  ---
  
  ### Answer by username_1 on 2019-04-23. Score: 0
  
  The weights of the filters do not always and necessarily decrease. Consider the extreme case when you initialise them to $-\infty$ and you want to approximate a function different than the one the CNN represents initially with all weights set to $-\infty$. You will have to increase one or more weights.
  
  
  ---
  
  