Answers.Body: <p>RNN is a deeply non-linear function over time, how the black linear line is deduced? </p>
  
  <p>Assuming you are doing just linear regression if the least square error was used as the loss function, it will have a probabilistic interpretation   </p>
  
  <p><span class="math-container">$$y^{(i)}|x^{(i)};\theta \sim \mathcal N(\theta^Tx^{(i)}, \sigma^2)$$</span></p>
  
  <p><span class="math-container">$Y$</span> is conditioned on <span class="math-container">$X$</span> parameterized by <span class="math-container">$\theta$</span>, with a Gaussian distribution, thus for every data point <span class="math-container">$x$</span>, there is a corresponding <span class="math-container">$y$</span> which if you are doing maximum likelihood estimation is just the mean of the Gaussian, hence variance is introduced to express the noise. if you are doing mini-batch is not guaranteed to reach a global minimum.</p>
  
  <p>Side note: If you normalize the data with min-max scaling, please make sure to use only the train set, if you include the dev/test set, you are doing some kind of data snooping, generalization error will be biased.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-07T07:02:26.253Z
Answers.Text: The black line is a reference 1:1 line. The blue dots are predictions vs. true values. I do use mini-batches. MSE is the loss function and standard adam is the optimizer. Implementation is done in Keras with TF backend. I'm not seeing clearly yet what your answer states?
  
  And yes, scaling parameters were calculated based on the training set only.
Answers.UserDisplayName: Kristof
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-07T07:17:32.827Z
Answers.Text: Umm, I see, normalize the inputs to have zero mean and unit variance instead of min-max scaling. augment the LSTM with Mixture Density Outputs as proposed in Generating Sequences With Recurrent Neural Networks (Grave-2013).
Answers.UserDisplayName: Fadi Bakoura
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-07T07:25:56.040Z
Answers.Text: Be careful of the output formulation, if you could define relative outputs instead of absolute real values, it will be way better.
Answers.UserDisplayName: Fadi Bakoura
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-18T14:37:16.233Z
Answers.Text: What do you mean by relative as opposed to absolute real values? min-max scaling or standard scaling was already included in the procedure.
Answers.UserDisplayName: Kristof
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-18T14:46:08.990Z
Answers.Text: Use the relative difference between subsequent timesteps instead of their corresponding absolute values, e.g. x2= x2-x1, don't forget to renormalize them to have zero mean unit variance
Answers.UserDisplayName: Fadi Bakoura
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-05T17:13:08.737Z
Answers.LastActivityDate: 2020-03-31T20:30:49.910Z
Answers.LastEditDate: 2020-03-31T20:30:49.910Z
Answers.UserDisplayName: Fadi Bakoura
Body: <p>I'm training an LSTM network with multiple inputs and several LSTM layers in order to set up a time series gap filling procedure. The LSTM is trained bidirectionally with &quot;tanh&quot; activation on the outputs of the LSTM, and one Dense layer with &quot;linear&quot; activation comes at the end to predict the outputs. The following scatterplot of real outputs vs the predictions illustrates the problem:</p>
  <p>Outputs (X-axis) vs predictions (Y-axis):</p>
  <p><a href="https://i.sstatic.net/Smj3i.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Smj3i.png" alt="image" /></a></p>
  <p>The network is definitely not performing too bad and I'll be updating the parameters in the next trials, but the issue at hand always reappears. The highest outputs are clearly underestimated, and the lowest values are overestimated, clearly systematic.</p>
  <p>I have tried min-max scaling on inputs and outputs and normalizing inputs and outputs, and the latter performs slightly better, but the issue persists.</p>
  <p>I've looked a lot in existing threads and Q&amp;As, but I haven't seen something similar.</p>
  <p>I'm wondering if anyone here sees this and immediately knows the possible cause (activation function? Preprocessing? Optimizer? Lack of weights during training? ... ?). Or, and in that case, it would also be good to know if this is impossible to find out without extensive testing.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-04T19:32:28.107Z
Comments.Text: Try using sigmoid layer in the last layer instead of linear activation.
Comments.UserDisplayName: thecomplexitytheorist
Tags: <neural-networks><python><keras><recurrent-neural-networks><long-short-term-memory>
Title: Over- and underestimations of the lowest and highest values in LSTM network
UserDisplayName: Kristof
ThreadText: # Over- and underestimations of the lowest and highest values in LSTM network
  
  (Asked by: username_0 on 2018-04-30)
  
  I'm training an LSTM network with multiple inputs and several LSTM layers in order to set up a time series gap filling procedure. The LSTM is trained bidirectionally with "tanh" activation on the outputs of the LSTM, and one Dense layer with "linear" activation comes at the end to predict the outputs. The following scatterplot of real outputs vs the predictions illustrates the problem:
  
  Outputs (X-axis) vs predictions (Y-axis):
  
  [![image](https://i.sstatic.net/Smj3i.png)](https://i.sstatic.net/Smj3i.png)
  
  The network is definitely not performing too bad and I'll be updating the parameters in the next trials, but the issue at hand always reappears. The highest outputs are clearly underestimated, and the lowest values are overestimated, clearly systematic.
  
  I have tried min-max scaling on inputs and outputs and normalizing inputs and outputs, and the latter performs slightly better, but the issue persists.
  
  I've looked a lot in existing threads and Q&As, but I haven't seen something similar.
  
  I'm wondering if anyone here sees this and immediately knows the possible cause (activation function? Preprocessing? Optimizer? Lack of weights during training? ... ?). Or, and in that case, it would also be good to know if this is impossible to find out without extensive testing.
  
  **Tags:** neural-networks, python, keras, recurrent-neural-networks, long-short-term-memory
  > **Comments:**
  > **username_1**: Try using sigmoid layer in the last layer instead of linear activation.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-05-05. Score: 1
  
  RNN is a deeply non-linear function over time, how the black linear line is deduced?
  
  Assuming you are doing just linear regression if the least square error was used as the loss function, it will have a probabilistic interpretation
  
  $$y^{(i)}|x^{(i)};\theta \sim \mathcal N(\theta^Tx^{(i)}, \sigma^2)$$
  
  $Y$ is conditioned on $X$ parameterized by $\theta$, with a Gaussian distribution, thus for every data point $x$, there is a corresponding $y$ which if you are doing maximum likelihood estimation is just the mean of the Gaussian, hence variance is introduced to express the noise. if you are doing mini-batch is not guaranteed to reach a global minimum.
  
  Side note: If you normalize the data with min-max scaling, please make sure to use only the train set, if you include the dev/test set, you are doing some kind of data snooping, generalization error will be biased.
  
  > **Comments:**
  > **username_0**: The black line is a reference 1:1 line. The blue dots are predictions vs. true values. I do use mini-batches. MSE is the loss function and standard adam is the optimizer. Implementation is done in Keras with TF backend. I'm not seeing clearly yet what your answer states?  And yes, scaling parameters were calculated based on the training set only.  
  > **username_2**: Umm, I see, normalize the inputs to have zero mean and unit variance instead of min-max scaling. augment the LSTM with Mixture Density Outputs as proposed in Generating Sequences With Recurrent Neural Networks (Grave-2013).  
  > **username_2**: Be careful of the output formulation, if you could define relative outputs instead of absolute real values, it will be way better.  
  > **username_0**: What do you mean by relative as opposed to absolute real values? min-max scaling or standard scaling was already included in the procedure.  
  > **username_2**: Use the relative difference between subsequent timesteps instead of their corresponding absolute values, e.g. x2= x2-x1, don't forget to renormalize them to have zero mean unit variance  
  
  ---
  
  