Answers.Body: <p>A friend of mine answered on this question in different social media on different language, I'll post his answer here:</p>
  <blockquote>
  <h3>1. scaler should be saved in this case.</h3>
  <p>You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be &quot;fit&quot; once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.</p>
  <h3>2. It's not a fact that this will break the model.</h3>
  <p>I think everything is clear here, but just in case:
  Let us consider the simplest case. Suppose there is a linear dependence <code>y = 3 * x</code>, we don't know about it, but there is a dataset:</p>
  <pre><code>X Y
  1 3
  2 6
  3 9
  </code></pre>
  <p>If we assume that the model has the form <code>y = a * x</code>, in the process of learning it turns out that<br />
  <code>a = 3</code> gives the best score and when suddenly <code>x = 100</code> appears, it has no problem mapping to 300, even though 100 is strongly out of dataset <code>(1,2,3)</code>.</p>
  <h3>3. You can check if this breaks the model now.</h3>
  <p>Suppose there is a chip that takes values <code>0, 1, 2, 3, 4, 5, 6, 7, 8 ...</code>
  You are afraid that after some time samples with values greater than the current maximum <code>(9, 10, 11, ...)</code> will appear and the model will behave strangely on them. You can try to &quot;simulate&quot; this situation.
  Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.</p>
  </blockquote>
  <p>Also, there is answer for the second question: <a href="https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value">https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-08T12:34:14.317Z
Answers.LastActivityDate: 2021-05-08T12:34:14.317Z
Answers.UserDisplayName: banderlog013
Body: <p>I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:</p>
  <ol>
  <li>outliers, but I could fight them or use RobustScaler instead.</li>
  <li>I am not sure that some features in my dataset include all possible ranges. Like I have max(feature_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.</li>
  <li>Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !&gt;100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.</li>
  </ol>
  <p>My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.</p>
  <p>How should I handle this?</p>
  <p>My concerns example:</p>
  <pre><code>import pandas as pd
  from sklearn.preprocessing import MinMaxScaler
  
  scaler = MinMaxScaler()
  
  #### like, initial state
  df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
  &quot;&quot;&quot; output:
     A    B
  0  1   10
  1  2   40
  2  3   60
  3  4   80
  4  5  100
  &quot;&quot;&quot;
  
  scaler.fit_transform(df1)
  &quot;&quot;&quot; output:
  array([[0.        , 0.        ],
         [0.25      , 0.33333333],
         [0.5       , 0.55555556],
         [0.75      , 0.77777778],
         [1.        , 1.        ]])
  &quot;&quot;&quot;
  
  #### new data arrived, while preprocessing is the same
  df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
  &quot;&quot;&quot; output:
      A    B
  0   1   10
  1   2   40
  2   3   60
  3   4   80
  4   5  100
  5  10  120
  6  10  140
  &quot;&quot;&quot;
  
  # now 5 in &quot;A&quot; scaled to 0.4 instead of 1, same in &quot;B&quot;
  scaler.fit_transform(df2)
  &quot;&quot;&quot; output:
  array([[0.        , 0.        ],
         [0.11111111, 0.23076923],
         [0.22222222, 0.38461538],
         [0.33333333, 0.53846154],
         [0.44444444, 0.69230769],
         [1.        , 0.84615385],
         [1.        , 1.        ]])
  &quot;&quot;&quot;
  </code></pre>
  <p><strong>PS:</strong> I've duplicated this question in different communities (question in ai got most of views):</p>
  <ul>
  <li><a href="https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data">https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data</a></li>
  <li><a href="https://datascience.stackexchange.com/questions/94095/normalization-of-possibly-not-fully-representative-data">https://datascience.stackexchange.com/questions/94095/normalization-of-possibly-not-fully-representative-data</a></li>
  </ul>
  
Tags: <data-preprocessing><sequence-modeling><normalisation>
Title: Normalization of possibly not fully representative data
UserDisplayName: banderlog013
ThreadText: # Normalization of possibly not fully representative data
  
  (Asked by: username_0 on 2021-05-06)
  
  I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:
  
  1. outliers, but I could fight them or use RobustScaler instead.
  2. I am not sure that some features in my dataset include all possible ranges. Like I have max(feature\_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.
  3. Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !>100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.
  
  My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.
  
  How should I handle this?
  
  My concerns example:
  
  ```
  import pandas as pd
  from sklearn.preprocessing import MinMaxScaler
  
  scaler = MinMaxScaler()
  
  #### like, initial state
  df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
  """ output:
     A    B
  0  1   10
  1  2   40
  2  3   60
  3  4   80
  4  5  100
  """
  
  scaler.fit_transform(df1)
  """ output:
  array([[0.        , 0.        ],
         [0.25      , 0.33333333],
         [0.5       , 0.55555556],
         [0.75      , 0.77777778],
         [1.        , 1.        ]])
  """
  
  #### new data arrived, while preprocessing is the same
  df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
  """ output:
      A    B
  0   1   10
  1   2   40
  2   3   60
  3   4   80
  4   5  100
  5  10  120
  6  10  140
  """
  
  # now 5 in "A" scaled to 0.4 instead of 1, same in "B"
  scaler.fit_transform(df2)
  """ output:
  array([[0.        , 0.        ],
         [0.11111111, 0.23076923],
         [0.22222222, 0.38461538],
         [0.33333333, 0.53846154],
         [0.44444444, 0.69230769],
         [1.        , 0.84615385],
         [1.        , 1.        ]])
  """
  
  ```
  
  **PS:** I've duplicated this question in different communities (question in ai got most of views):
  
  * <https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data>
  * <https://datascience.stackexchange.com/questions/94095/normalization-of-possibly-not-fully-representative-data>
  
  **Tags:** data-preprocessing, sequence-modeling, normalisation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-05-08. Score: 1
  
  A friend of mine answered on this question in different social media on different language, I'll post his answer here:
  
  > ### 1. scaler should be saved in this case.
  >
  > You do fit\_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.
  >
  > ### 2. It's not a fact that this will break the model.
  >
  > I think everything is clear here, but just in case:
  > Let us consider the simplest case. Suppose there is a linear dependence `y = 3 * x`, we don't know about it, but there is a dataset:
  >
  > ```
  > X Y
  > 1 3
  > 2 6
  > 3 9
  >
  > ```
  >
  > If we assume that the model has the form `y = a * x`, in the process of learning it turns out that  
  > `a = 3` gives the best score and when suddenly `x = 100` appears, it has no problem mapping to 300, even though 100 is strongly out of dataset `(1,2,3)`.
  >
  > ### 3. You can check if this breaks the model now.
  >
  > Suppose there is a chip that takes values `0, 1, 2, 3, 4, 5, 6, 7, 8 ...`
  > You are afraid that after some time samples with values greater than the current maximum `(9, 10, 11, ...)` will appear and the model will behave strangely on them. You can try to "simulate" this situation.
  > Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.
  
  Also, there is answer for the second question: <https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value>
  
  
  ---
  
  