Answers.Body: <p>I found the issue, I should have done more unit testing. Upon computing the batch loss before backpropagation, one of the dimension of the "prediction" tensor was not corresonding to the "truth" tensor. The shape match but the content is not the one that was supposed to be.
  This is due to how the NLL loss is implemented in pytorch which I was not aware of ... </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-20T19:48:23.957Z
Answers.LastActivityDate: 2020-02-20T19:48:23.957Z
Answers.UserDisplayName: Johncowk
Body: <p>I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in <code>[0,13]</code>. I have noticed that one-hot encoding it and using cross-entropy loss gives better results than MSE loss. </p>
  
  <p>Here is my problem : no matter how deep I make the network or how many fully connected layers I add I always obtain pretty much the same behavior. Changing the optimizer also doesn't really help.</p>
  
  <ol>
  <li>The loss function quickly decreases then stagnates with a very high variance and never goes down again.</li>
  <li>The prediction seems to be offset around the value <code>9</code>, I really do not understand why since I have one-hot encoded the input and the output.</li>
  </ol>
  
  <p>Here is an example of a the results of a typical training phase, with the total loss :
  <a href="https://i.sstatic.net/o51R3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/o51R3.png" alt="enter image description here"></a><a href="https://i.sstatic.net/0id4C.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0id4C.png" alt="enter image description here"></a></p>
  
  <p>Do you have any tips/ideas as to how I could improve this or could have gone wrong ? I am a bit of a beginner in ML si I might have missed something. I can also include the code (in PyTorch) if necessary.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-19T13:41:28.257Z
Comments.Text: What is the size of your data? And the number of trainable parameters in your network?
Comments.UserDisplayName: Sharan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-19T14:19:35.630Z
Comments.Text: The dataset is quite small (about 200 time-series of 50 points each). The number of trainable parameter depends, as said, I have tried various number of layers and hidden sizes but it didn't change that much.
Comments.UserDisplayName: Johncowk
Tags: <long-short-term-memory><pytorch><time-series><attention>
Title: time-series prediction : loss going down, then stagnates with very high variance
UserDisplayName: Johncowk
ThreadText: # time-series prediction : loss going down, then stagnates with very high variance
  
  (Asked by: username_0 on 2020-02-19)
  
  I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in `[0,13]`. I have noticed that one-hot encoding it and using cross-entropy loss gives better results than MSE loss.
  
  Here is my problem : no matter how deep I make the network or how many fully connected layers I add I always obtain pretty much the same behavior. Changing the optimizer also doesn't really help.
  
  1. The loss function quickly decreases then stagnates with a very high variance and never goes down again.
  2. The prediction seems to be offset around the value `9`, I really do not understand why since I have one-hot encoded the input and the output.
  
  Here is an example of a the results of a typical training phase, with the total loss :
  [![enter image description here](https://i.sstatic.net/o51R3.png)](https://i.sstatic.net/o51R3.png)[![enter image description here](https://i.sstatic.net/0id4C.png)](https://i.sstatic.net/0id4C.png)
  
  Do you have any tips/ideas as to how I could improve this or could have gone wrong ? I am a bit of a beginner in ML si I might have missed something. I can also include the code (in PyTorch) if necessary.
  
  **Tags:** long-short-term-memory, pytorch, time-series, attention
  > **Comments:**
  > **username_1**: What is the size of your data? And the number of trainable parameters in your network?  
  > **username_0**: The dataset is quite small (about 200 time-series of 50 points each). The number of trainable parameter depends, as said, I have tried various number of layers and hidden sizes but it didn't change that much.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-02-20. Score: 1
  
  I found the issue, I should have done more unit testing. Upon computing the batch loss before backpropagation, one of the dimension of the "prediction" tensor was not corresonding to the "truth" tensor. The shape match but the content is not the one that was supposed to be.
  This is due to how the NLL loss is implemented in pytorch which I was not aware of ...
  
  
  ---
  
  