Body: <p>The specific problem I have is learning the relation <span class="math-container">$x^2$</span>. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to <span class="math-container">$19^2$</span>=361.</p>
  
  <p>I have the following LSTM architecture:</p>
  
  <pre><code>1 input node
  1 output node
  32 hidden units
  </code></pre>
  
  <p>Now, interestingly, I accidentally trained my network wrong, in that I forgot to reverse the expected output list when training. So I trained the network for:
  <span class="math-container">$$0 \rightarrow 361 \\ 1 \rightarrow 324 \\ 2 \rightarrow 289 \\ 3 \rightarrow 256 \\ ... \\ 17 \rightarrow 4 \\ 18 \rightarrow 1 \\ 19 \rightarrow 0$$</span></p>
  
  <p>Starting with a learning rate of 1 and halving it every 400 epochs, after 3000 epochs my error (which started somwhere in the millions) was 0.2.</p>
  
  <p>However, when I went to correct this mistake, my error will hardly ever go beneath 100,000. Testing the network shows it does well in the low inputs, but once it starts to get to <span class="math-container">$16^2$</span> onwards, it really struggles to increase the output values past ~250.</p>
  
  <p>I was just wondering if anyone has an explanation for this, as to why the LSTM struggles to learn to increase exponentially but seems to be able to learn to decrease exponentially just fine.</p>
  
  <p>EDIT WITH CODE:</p>
  
  <pre><code>a = np.array([i for i in range(20)])
  b = np.array([i**2 for i in range(20)])
  np.random.seed(5)
  ls = LSTM(1, 1, 32, learning_rate=1, regression=1)
  # Input size = 1, output size = 1, hidden units = 32
  if 1:
      for k in range(3000):
          total = 0
          for i in range(20):
              ls * a[i]
          for i in range(20):
              total += ls / b[i]
          if k % 400 == 0:
              ls.learning_rate *= 0.5
          print(k, ":", total)
          ls.update_params()
  for i in a:
      print(i, ls*i)
  #ls.save_state('1.trainstate')
  for i in range(20,30):
      print(ls*i)
  </code></pre>
  
  <p>Note this code uses a class I wrote using numpy. If wanted I'll include this code as well it's just that is ~300 lines and I don't expect anyone to go through all that</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-14T10:03:24.460Z
Comments.Text: I suspect that it is a problem with vanishing gradients, something like: If you feed in 19, you saturate the tanh in the LSTM, driving it to the regions where the gradient is almost 0, which is more hurtful when the target is 361 then when the target is 0 (since your output at the beginning is 0). But this is difficult to verify without having access to the model. Have you looked at the gradients in tensorboard? And could you post the mapping after training? I would also suggest to train the model again, but this time scale input and output to 0 to 1, and see if this observation still holds.
Comments.UserDisplayName: Luca Thiede
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-17T01:35:53.807Z
Comments.Text: @LucaThiede That's a really good suggestion - it makes intuitive sense. I'll have to run the model again with scaled input/output. I'll try and create some form of visualisation of the gradients at each step
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-17T18:44:13.890Z
Comments.Text: @Recessive First LSTM shouldnt be used for this type of problem as its not temporal, second I ran this and got same loss for both ways (for me using MSE and same lr scheduler I get ~28000 loss), share your code and let us see if we can reproduce your behavior
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-18T02:37:59.463Z
Comments.Text: @mshlis I know it's not ideal for LSTMs, the question is more of just curiosity than anything. I've included my code in the edit
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-18T12:21:20.683Z
Comments.Text: Do you apply weight regularization? That would make difficult to output high values, even more if input values are low.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-18T14:47:30.230Z
Comments.Text: @David By weight regularization do you mean something like weight decay? In which case no, I don't
Comments.UserDisplayName: Recessive
Tags: <neural-networks><long-short-term-memory>
Title: Why do regression LSTMs learn high to low inputs significantly better than low to high?
UserDisplayName: Recessive
ThreadText: # Why do regression LSTMs learn high to low inputs significantly better than low to high?
  
  (Asked by: username_0 on 2019-11-26)
  
  The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361.
  I have the following LSTM architecture:
  1 input node
  1 output node
  32 hidden units
  
  Now, interestingly, I accidentally trained my network wrong, in that I forgot to reverse the expected output list when training. So I trained the network for:
  $$0 \rightarrow 361 \\ 1 \rightarrow 324 \\ 2 \rightarrow 289 \\ 3 \rightarrow 256 \\ ... \\ 17 \rightarrow 4 \\ 18 \rightarrow 1 \\ 19 \rightarrow 0$$
  Starting with a learning rate of 1 and halving it every 400 epochs, after 3000 epochs my error (which started somwhere in the millions) was 0.2.
  However, when I went to correct this mistake, my error will hardly ever go beneath 100,000. Testing the network shows it does well in the low inputs, but once it starts to get to $16^2$ onwards, it really struggles to increase the output values past ~250.
  I was just wondering if anyone has an explanation for this, as to why the LSTM struggles to learn to increase exponentially but seems to be able to learn to decrease exponentially just fine.
  EDIT WITH CODE:
  a = np.array([i for i in range(20)])
  b = np.array([i**2 for i in range(20)])
  np.random.seed(5)
  ls = LSTM(1, 1, 32, learning_rate=1, regression=1)
  # Input size = 1, output size = 1, hidden units = 32
  if 1:
      for k in range(3000):
          total = 0
          for i in range(20):
              ls * a[i]
          for i in range(20):
              total += ls / b[i]
          if k % 400 == 0:
              ls.learning_rate *= 0.5
          print(k, ":", total)
          ls.update_params()
  for i in a:
      print(i, ls*i)
  #ls.save_state('1.trainstate')
  for i in range(20,30):
      print(ls*i)
  
  Note this code uses a class I wrote using numpy. If wanted I'll include this code as well it's just that is ~300 lines and I don't expect anyone to go through all that
  
  **Tags:** neural-networks, long-short-term-memory
  > **Comments:**
  > **username_1**: I suspect that it is a problem with vanishing gradients, something like: If you feed in 19, you saturate the tanh in the LSTM, driving it to the regions where the gradient is almost 0, which is more hurtful when the target is 361 then when the target is 0 (since your output at the beginning is 0). But this is difficult to verify without having access to the model. Have you looked at the gradients in tensorboard? And could you post the mapping after training? I would also suggest to train the model again, but this time scale input and output to 0 to 1, and see if this observation still holds.  
  > **username_0**: @LucaThiede That's a really good suggestion - it makes intuitive sense. I'll have to run the model again with scaled input/output. I'll try and create some form of visualisation of the gradients at each step  
  > **username_2**: @Recessive First LSTM shouldnt be used for this type of problem as its not temporal, second I ran this and got same loss for both ways (for me using MSE and same lr scheduler I get ~28000 loss), share your code and let us see if we can reproduce your behavior  
  > **username_0**: @mshlis I know it's not ideal for LSTMs, the question is more of just curiosity than anything. I've included my code in the edit  
  > **username_3**: Do you apply weight regularization? That would make difficult to output high values, even more if input values are low.  
  > **username_0**: @David By weight regularization do you mean something like weight decay? In which case no, I don't  
  
  ---
  
  ## 0 answers
  
  