Answers.Body: <p>Vanishing gradient is a common problem in RNN.</p>
  
  <p>A common way to deal with it is the method of gradient clipping (mainly you define a maximum and/ or a minimum threshold). <a href="https://www.quora.com/What-is-gradient-clipping-and-why-is-it-necessary" rel="nofollow noreferrer">see here for more information</a> </p>
  
  <p>Further information and piece of code to implement it can be found in <a href="https://stackoverflow.com/questions/36498127/how-to-apply-gradient-clipping-in-tensorflow">SO here</a></p>
  
  <p>Hope it helps ! </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-13T01:32:15.557Z
Answers.LastActivityDate: 2018-06-13T01:32:15.557Z
Answers.UserDisplayName: nsaura
Answers.Body: <p>I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-13T17:45:41.863Z
Answers.Text: It means that I normalize the input for the network layers. I hope this explains.
Answers.UserDisplayName: Tom Z
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-11T05:28:32.173Z
Answers.LastActivityDate: 2018-04-11T05:28:32.173Z
Answers.UserDisplayName: Tom Z
Body: <p>When training on large neural network, how to deal with the case that the gradients are too small to have any impact?</p>
  
  <p>FYI, I have an RNN, which has multiple LSTM cells and each cell has hundreds of neurons. Each training data has thousands of steps, so the RNN would unroll thousands of times. When I print out all gradients, they are very small, like e-20 of the variable values. Therefore the training does not change the variable values at all. </p>
  
  <p>BTW, I think this is not an issue of vanishing gradients. Note that the gradients are uniformly small from the beginning to the end.</p>
  
  <p>Any suggestion to overcome this issue?</p>
  
  <p>Thank!</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-09T16:28:21.940Z
Comments.Text: Did you tried Backprogation Through Time  (BPTT) ? Need more information about your net and task to help you
Comments.UserDisplayName: user3352632
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-11T05:29:23.773Z
Comments.Text: Thanks for your comment and suggestion. I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network.
Comments.UserDisplayName: Tom Z
Tags: <gaming>
Title: Too small gradient on large neural network
UserDisplayName: Tom Z
ThreadText: # Too small gradient on large neural network
  
  (Asked by: username_0 on 2018-04-06)
  
  When training on large neural network, how to deal with the case that the gradients are too small to have any impact?
  
  FYI, I have an RNN, which has multiple LSTM cells and each cell has hundreds of neurons. Each training data has thousands of steps, so the RNN would unroll thousands of times. When I print out all gradients, they are very small, like e-20 of the variable values. Therefore the training does not change the variable values at all.
  
  BTW, I think this is not an issue of vanishing gradients. Note that the gradients are uniformly small from the beginning to the end.
  
  Any suggestion to overcome this issue?
  
  Thank!
  
  **Tags:** gaming
  
  > **Comments:**
  > **username_1**: Did you tried Backprogation Through Time  (BPTT) ? Need more information about your net and task to help you  
  > **username_0**: Thanks for your comment and suggestion. I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-06-13. Score: 3
  
  Vanishing gradient is a common problem in RNN.
  
  A common way to deal with it is the method of gradient clipping (mainly you define a maximum and/ or a minimum threshold). [see here for more information](https://www.quora.com/What-is-gradient-clipping-and-why-is-it-necessary)
  
  Further information and piece of code to implement it can be found in [SO here](https://stackoverflow.com/questions/36498127/how-to-apply-gradient-clipping-in-tensorflow)
  
  Hope it helps !
  
  
  ---
  
  ### Answer by username_0 on 2018-04-11. Score: 2
  
  I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network.
  
  > **Comments:**
  > **username_0**: It means that I normalize the input for the network layers. I hope this explains.  
  
  ---
  
  