Body: <p>I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of <a href="https://stats.stackexchange.com/q/126238/82135">RELU advantages</a> on faster computation and no <strong>vanishing gradient</strong> problem. But about vanishing gradient, the main problem is about the backpropagation algorithm going to zero quickly if using sigmoid or tanh. So I would like to try to compensate this effect that affects deep layers with a variable <code>learning rate</code> for every layer, increasing the coefficient every time you go a layer deeper to compensate the vanishing gradient.</p>
  <p>I have read about <a href="https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks" rel="nofollow noreferrer">adaptive learning rate</a>, but it seems to refer to a technique to change the learning rate on every epoch, I'm looking for a different learning rate for every layer, into any epoch.</p>
  <ol>
  <li><p>Based on your experience, do you think that is a good effort to try?</p>
  </li>
  <li><p>Do you know some libraries I can use that already let you define the learning rate as a function and not a constant?</p>
  </li>
  <li><p>If such function exists, it will be better to define a simple function <code>lr=(a*n)*0.001</code> where <code>n</code> is layer number, and <code>a</code> a multiplier based on experience, of we will need the inverse of the activation function to compensate enough the gradient vanishing?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T11:49:04.763Z
Comments.Text: I see two possibilities: (a) copy and modify an existing optimizer like SGD or Adam to do what you want, (b) use SGD with lr=1 and multiply the gradients of each layer by whatever learning rate you want (this is mathematically equivalent to what you want).
Comments.UserDisplayName: Ricardo Cruz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T14:28:18.130Z
Comments.Text: I believe you might run into issues related to errors inherent in numerical computation. That is, the calculation of the gradient for these "compensated" layers might become increasingly inaccurate and prevent learning from occurring. This would be mostly true in deep networks because the compensated learning rate might result in very large numbers multiplied by very small numbers (a known challenge in numerical computation). It would be interesting if someone had insights into this.
Comments.UserDisplayName: respectful
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T18:37:35.450Z
Comments.Text: Alternative idea: normalize each layer's gradient and scale by normalized gradient multiplier `ngm`, which is somewhat similar to learning rate.
Comments.UserDisplayName: ShadowsInRain
Tags: <deep-learning><backpropagation><activation-functions><learning-rate><vanishing-gradient-problem>
Title: Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?
UserDisplayName: Rogelio Triviño
ThreadText: # Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?
  
  (Asked by: username_0 on 2020-08-06)
  
  I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of [RELU advantages](https://stats.stackexchange.com/q/126238/82135) on faster computation and no **vanishing gradient** problem. But about vanishing gradient, the main problem is about the backpropagation algorithm going to zero quickly if using sigmoid or tanh. So I would like to try to compensate this effect that affects deep layers with a variable `learning rate` for every layer, increasing the coefficient every time you go a layer deeper to compensate the vanishing gradient.
  
  I have read about [adaptive learning rate](https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks), but it seems to refer to a technique to change the learning rate on every epoch, I'm looking for a different learning rate for every layer, into any epoch.
  
  1. Based on your experience, do you think that is a good effort to try?
  2. Do you know some libraries I can use that already let you define the learning rate as a function and not a constant?
  3. If such function exists, it will be better to define a simple function `lr=(a*n)*0.001` where `n` is layer number, and `a` a multiplier based on experience, of we will need the inverse of the activation function to compensate enough the gradient vanishing?
  
  **Tags:** deep-learning, backpropagation, activation-functions, learning-rate, vanishing-gradient-problem
  
  > **Comments:**
  > **username_1**: I see two possibilities: (a) copy and modify an existing optimizer like SGD or Adam to do what you want, (b) use SGD with lr=1 and multiply the gradients of each layer by whatever learning rate you want (this is mathematically equivalent to what you want).  
  > **username_2**: I believe you might run into issues related to errors inherent in numerical computation. That is, the calculation of the gradient for these "compensated" layers might become increasingly inaccurate and prevent learning from occurring. This would be mostly true in deep networks because the compensated learning rate might result in very large numbers multiplied by very small numbers (a known challenge in numerical computation). It would be interesting if someone had insights into this.  
  > **username_3**: Alternative idea: normalize each layer's gradient and scale by normalized gradient multiplier `ngm`, which is somewhat similar to learning rate.  
  
  ---
  
  ## 0 answers
  
  