Answers.Body: <p>The problem as you already pointed out is in the high change variance of the new feature. Normalising using the mean does not impact the variance since it's merely a linear transformation, still good to keep all feature in the same range but not impacting the variance at all.</p>
  <p>What you need is a non linear transformation that change the magnitude in change variance without introducing artefacts. The most simple thing to try would be to just take the log of the new feature. Since you're working with time series another possibility could be applying a moving average, exploring different time windows to see how much the feature will be smoothed out.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T23:26:26.963Z
Answers.Text: Thanks! If I used log as a normalization would it not generally "suppress" the sensitivity of the model to the changes of that feature? Or is that the tradeoff I have to accept?
Answers.UserDisplayName: Darren Rahnemoon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T23:28:48.947Z
Answers.Text: Also is there any point for choosing a specific base of the log function given the variance or just better off with natural log?
Answers.UserDisplayName: Darren Rahnemoon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T19:36:35.753Z
Answers.LastActivityDate: 2023-04-12T19:36:35.753Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>In a time series regression problem I'm predicting &quot;change&quot; rather than the actual intended value i.e
  Instead of:</p>
  <pre><code>time, feature 1, feature 2, ....
  2020, 1, 3.3
  2021, 1.5, 5.2
  2022, 1.3, 6.1
  now, y_pred_1, y_pred_2, ....
  </code></pre>
  <p>I'm feeding the model and predicting percent changes:</p>
  <pre><code>2020 0, 0
  2021 0.5, 0.57 # 5.2 / 3.3 - 1 = 0.57 percent change
  2022, -0.13, 0.17
  </code></pre>
  <p>Now a problem that I'm facing is that when I introduce a new feature of a much larger magnitude and variance my model diverges during training:</p>
  <pre><code># new feature
  2020, 3000
  2021 4
  2022 600
  
  # Which is transformed to percent changes
  2020, 0
  2021, 0.99
  2022, 149 # oof
  </code></pre>
  <p>I tried mean normalizing that particular feature before transforming it to the percent change domain but it still diverges.
  Any ideas what else I can try to introduce this new feature to the inputs? What are some best practices for such contrasted inputs?</p>
  <p>To get a general idea of the model if it has something to do with this:</p>
  <p>The model consists of 5 parallel flows that get added together at the end. Each flow has two pairs of CNNs with different kernel sizes followed by 1-3 LSTMs. The output shape of each flow is the same so they can be added together. Then after adding it's followed by two fully connected layers predicting the change for the given features.</p>
  
Tags: <long-short-term-memory><keras><time-series><regression>
Title: Regression Model diverging after adding a new feature with higher variance and magnitude
UserDisplayName: Darren Rahnemoon
ThreadText: # Regression Model diverging after adding a new feature with higher variance and magnitude
  
  (Asked by: username_0 on 2023-04-12)
  
  In a time series regression problem I'm predicting "change" rather than the actual intended value i.e
  Instead of:
  
  ```
  time, feature 1, feature 2, ....
  2020, 1, 3.3
  2021, 1.5, 5.2
  2022, 1.3, 6.1
  now, y_pred_1, y_pred_2, ....
  
  ```
  
  I'm feeding the model and predicting percent changes:
  
  ```
  2020 0, 0
  2021 0.5, 0.57 # 5.2 / 3.3 - 1 = 0.57 percent change
  2022, -0.13, 0.17
  
  ```
  
  Now a problem that I'm facing is that when I introduce a new feature of a much larger magnitude and variance my model diverges during training:
  
  ```
  # new feature
  2020, 3000
  2021 4
  2022 600
  
  # Which is transformed to percent changes
  2020, 0
  2021, 0.99
  2022, 149 # oof
  
  ```
  
  I tried mean normalizing that particular feature before transforming it to the percent change domain but it still diverges.
  Any ideas what else I can try to introduce this new feature to the inputs? What are some best practices for such contrasted inputs?
  
  To get a general idea of the model if it has something to do with this:
  
  The model consists of 5 parallel flows that get added together at the end. Each flow has two pairs of CNNs with different kernel sizes followed by 1-3 LSTMs. The output shape of each flow is the same so they can be added together. Then after adding it's followed by two fully connected layers predicting the change for the given features.
  
  **Tags:** long-short-term-memory, keras, time-series, regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-12. Score: 1 | Accepted answer
  
  The problem as you already pointed out is in the high change variance of the new feature. Normalising using the mean does not impact the variance since it's merely a linear transformation, still good to keep all feature in the same range but not impacting the variance at all.
  
  What you need is a non linear transformation that change the magnitude in change variance without introducing artefacts. The most simple thing to try would be to just take the log of the new feature. Since you're working with time series another possibility could be applying a moving average, exploring different time windows to see how much the feature will be smoothed out.
  
  > **Comments:**
  > **username_0**: Thanks! If I used log as a normalization would it not generally "suppress" the sensitivity of the model to the changes of that feature? Or is that the tradeoff I have to accept?  
  > **username_0**: Also is there any point for choosing a specific base of the log function given the variance or just better off with natural log?  
  
  ---
  
  