title: Should I use MinMaxScaler which was fit on train dataset to transform test dataset, or use a separate MinMaxScaler to fit and transform test dataset?
body: <p>Assume that I have 3 dataset in a ML problem.</p>
  <p><code>train dataset</code>: used to estimate ML model parameters (training)</p>
  <p><code>test dataset</code>: used to evaulate trained model, calculate accuracy of trained model</p>
  <p><code>prediction dataset</code>: used only for prediction after model deployment</p>
  <p>I don't have <code>evaluation dataset</code>, and I use <strong>Grid Search</strong> with k-fold cross validation to find the best model.</p>
  <p>Also, I have two python scripts as follows:</p>
  <p><code>train.py</code>: used to train and test ML model, load train and test dataset, save the trained model, best model is found by Grid Search.</p>
  <p><code>predict.py</code>: used to load pre-trained model &amp; load prediction dataset, predict model output and calculate accuracy.</p>
  <p>Before starting training process in <code>train.py</code>, I use <strong>MinMaxScaler</strong> as follows:</p>
  <pre><code>from sklearn.preprocessing import MinMaxScaler
  scaler = MinMaxScaler()
  
  scaler.fit(x_train)   # fit only on train dataset
  x_train_norm = scaler.transform(x_train)
  x_test_norm = scaler.transform(x_test)
  </code></pre>
  <p>In <code>predict.py</code>, after loding prediction dataset, I need to use the same data pre-processing as below:</p>
  <pre><code>from sklearn.preprocessing import MinMaxScaler
  scaler = MinMaxScaler()
  
  scaler.fit(x_predict)   
  x_predict_norm = scaler.transform(x_predict)
  </code></pre>
  <p>As you can see above, both fit and transform are done on prediction dataset. However, in <code>train.py</code>, fit is done on train dataset, and the same MinMaxScaler is applied to transform test dataset.</p>
  <p>My understanding is that test dataset is a simulation of real data that model is supposed to predict after deployment. Therefore, data pre-processing of test and prediction dataset should be the same.</p>
  <p>I think separate MinMaxScaler should be used in <code>train.py</code> for train and test dataset as follows:</p>
  <pre><code>from sklearn.preprocessing import MinMaxScaler
  scaler_train = MinMaxScaler()
  scaler_test = MinMaxScaler()
  
  scaler_train.fit(x_train)   # fit only on train dataset
  x_train_norm = scaler_train.transform(x_train)
  
  scaler_test.fit(x_test)   # fit only on test dataset
  x_test_norm = scaler_test.transform(x_test)
  </code></pre>
  <p>What is the difference?</p>
  <p>Value of <code>x_test_norm</code> will be different if I use separate MinMaxScaler as explained above. In this case, value of <code>x_test_norm</code> is in the range of [-1, 1]. However, If I transform test dataset by a MinMaxScaler which was fit by train dataset, value of <code>x_test_norm</code> can be outside the range of [-1, 1].</p>
  <p>Please let me know your idea about it.</p>