Answers.Body: <p>What may be more informative in terms of whether it is learning or not, is to track gradients.</p>
  <p>Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.</p>
  <p>You are developing with tf.keras, in which case you can add to your tensorboard callback:
  <code>tf.keras.callbacks.TensorBoard(write_grads=True)</code></p>
  <p>Additional experiments may include:</p>
  <ul>
  <li>trying shorter length sequences and compare grad flow</li>
  <li>try replacing <code>tanh</code> with alternative activation functions in the LSTM layers of your model <code>tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu())</code> (see <code>tanh</code> saddle points problem)</li>
  </ul>
  <p><a href="https://i.sstatic.net/6DoLf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6DoLf.png" alt="enter image description here" /></a></p>
  <p>(graph borrowed from <a href="https://d2l.ai/chapter_optimization/optimization-intro.html?highlight=saddle%20points" rel="nofollow noreferrer">d2l</a>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T10:16:27.387Z
Answers.Text: It seems that `tf.keras.callbacks.TensorBoard(write_grads=True)` is deprecated in tensorflow 2.4 but I found this [article](https://medium.com/@leenabora1/how-to-keep-a-track-of-gradients-vanishing-exploding-gradients-b0bbaa1dcb93) on how to do it manually. Also I'll try the leaky relu too, thanks.
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-15T10:31:02.643Z
Answers.Text: Oh interesting article @SepehrGolestanian, wasn't aware. I have been using tbX for pytorch and it seems to still be there. Feel free to upvote/choose as solution if answer was useful.
Answers.UserDisplayName: hH1sG0n3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-15T12:50:37.480Z
Answers.Text: I’ve tried leaky relu too but nothing changed
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-15T12:57:15.607Z
Answers.Text: You mean nothing changed in the gradients or the weigh/bias distributions?
Answers.UserDisplayName: hH1sG0n3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-15T14:16:44.847Z
Answers.Text: I don't know anyway to check gradients in tensorflow but the weight distributions didn't change.
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-19T14:35:39.297Z
Answers.Text: It is difficult to know if you cannot. The weight distribution of a neural network that has been trained with regularisation is expected to be distributed normally. See here relevant paper. http://proceedings.mlr.press/v37/blundell15.pdf
Answers.UserDisplayName: hH1sG0n3
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T15:52:32.070Z
Answers.LastActivityDate: 2021-03-12T16:03:47.337Z
Answers.LastEditDate: 2021-03-12T16:03:47.337Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:
  <a href="https://i.sstatic.net/YZ3Pw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YZ3Pw.png" alt="enter image description here" /></a></p>
  <p>I don't understand. Is this layer learning anything? Or no?</p>
  <p>Update: I've also tried LeakyReLU activation and also removed l2 regularization and this is what I got:
  <a href="https://i.sstatic.net/p3ZFK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/p3ZFK.png" alt="enter image description here" /></a>
  So I guess my layer isn't learning or does take more epochs to train LSTM layers? The gradients are not vanishing because the CNN layer before this is changing.</p>
  
Tags: <tensorflow><long-short-term-memory><weights>
Title: Is this LSTM layer learning anything?
UserDisplayName: Sepehr Golestanian
ThreadText: # Is this LSTM layer learning anything?
  
  (Asked by: username_0 on 2021-03-12)
  
  I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:
  [![enter image description here](https://i.sstatic.net/YZ3Pw.png)](https://i.sstatic.net/YZ3Pw.png)
  
  I don't understand. Is this layer learning anything? Or no?
  
  Update: I've also tried LeakyReLU activation and also removed l2 regularization and this is what I got:
  [![enter image description here](https://i.sstatic.net/p3ZFK.png)](https://i.sstatic.net/p3ZFK.png)
  So I guess my layer isn't learning or does take more epochs to train LSTM layers? The gradients are not vanishing because the CNN layer before this is changing.
  
  **Tags:** tensorflow, long-short-term-memory, weights
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-12. Score: 1
  
  What may be more informative in terms of whether it is learning or not, is to track gradients.
  
  Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.
  
  You are developing with tf.keras, in which case you can add to your tensorboard callback:
  `tf.keras.callbacks.TensorBoard(write_grads=True)`
  
  Additional experiments may include:
  
  * trying shorter length sequences and compare grad flow
  * try replacing `tanh` with alternative activation functions in the LSTM layers of your model `tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu())` (see `tanh` saddle points problem)
  
  [![enter image description here](https://i.sstatic.net/6DoLf.png)](https://i.sstatic.net/6DoLf.png)
  
  (graph borrowed from [d2l](https://d2l.ai/chapter_optimization/optimization-intro.html?highlight=saddle%20points))
  
  > **Comments:**
  > **username_0**: It seems that `tf.keras.callbacks.TensorBoard(write_grads=True)` is deprecated in tensorflow 2.4 but I found this [article](https://medium.com/@leenabora1/how-to-keep-a-track-of-gradients-vanishing-exploding-gradients-b0bbaa1dcb93) on how to do it manually. Also I'll try the leaky relu too, thanks.  
  > **username_1**: Oh interesting article @SepehrGolestanian, wasn't aware. I have been using tbX for pytorch and it seems to still be there. Feel free to upvote/choose as solution if answer was useful.  
  > **username_0**: I’ve tried leaky relu too but nothing changed  
  > **username_1**: You mean nothing changed in the gradients or the weigh/bias distributions?  
  > **username_0**: I don't know anyway to check gradients in tensorflow but the weight distributions didn't change.  
  > **username_1**: It is difficult to know if you cannot. The weight distribution of a neural network that has been trained with regularisation is expected to be distributed normally. See here relevant paper. http://proceedings.mlr.press/v37/blundell15.pdf  
  
  ---
  
  