Answers.Body: <p>Looking at your code, one set of data transformations were applied to the train data and a different set of transformations were applied to the test data. Different data transformations could account for different evaluation metric performance. </p>
  
  <p>It is best practices to put all data transformations in a function so they can be applied to all data in a similar way. </p>
  
  <p>Since you are using scikit-learn, <code>sklearn.compose.ColumnTransformer</code> is designed for this purpose. Example code for the Titanic dataset is <a href="https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T14:12:24.573Z
Answers.Text: I will try that but functionally the data transformation for both datasets is the same with the only difference being that the test data does not contain a "Ms" at factor level. Nevertheless you never know what makes a quirk in the data so I will package all transformations in a function and apply the same to both sets to eliminate this factor.
Answers.UserDisplayName: Fnguyen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-05T14:19:23.937Z
Answers.Text: I have cleaned up my code using your answer and a notebook I found on Kaggle as inspiration. By joining test and train dataset first, then doing all necessary transformations and only after splitting them again for modeling and prediction I was able to remove the error. They key to his was actually that I was applying an Imputation Method, I will describe that in an answer in detail.
Answers.UserDisplayName: Fnguyen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T14:03:07.587Z
Answers.LastActivityDate: 2020-02-03T14:03:07.587Z
Answers.UserDisplayName: Brian Spiering
Answers.Body: <p>@BrianSpiering was generally  correct in pointing out that you should always apply the same transformations to your train and test dataset.</p>
  
  <p>This was the key to my solution which was a bit more specific but might actually help others who encounter a similar problem.</p>
  
  <p>Specifically my mistake came about because of <strong>Imputation</strong>! Some of the factors I used for my model were <code>NA</code> in both the train and test data set. To complete the data I simply imputed these missings using mean and mode respectively. However since I did those transformations separately on both sets the actual mean/mode value that was used differed heavily! By applying the imputation on the full data I also imputed the same data for all missing cases which solved my error.</p>
  
  <p>My resulting accuracy in the public leaderboard is now at 74.2% which is fairly close to my local test score of 79.6%.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-05T14:23:36.780Z
Answers.LastActivityDate: 2020-02-05T14:23:36.780Z
Answers.UserDisplayName: Fnguyen
Body: <p>I have trained a XGboost model to predict survival for the <a href="https://www.kaggle.com/c/titanic/overview" rel="nofollow noreferrer">Kaggle Titanic ML competition</a>.</p>
  
  <p>As with all Kaggle competitions there is a <code>train</code> dataset with the target variable included and a <code>test</code> dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.</p>
  
  <p><strong>My problem:</strong></p>
  
  <p>I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the <code>train</code> data. The <strong>accuracy</strong> I get from this <strong>validation is ~80%</strong> which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).</p>
  
  <p>The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.</p>
  
  <p>However when I submit my predictions for the <code>test</code> set my <strong>public score is ~35%</strong> which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.</p>
  
  <p><strong>Why is my model so much worse on the <code>test</code>?</strong></p>
  
  <p>I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.</p>
  
  <p>However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.</p>
  
  <p><strong>So can you help me understand how those two issues happen at the same time:</strong></p>
  
  <ul>
  <li>Drastic difference between local accuracy and public score</li>
  <li>Reversing actually leads to the better public score.</li>
  </ul>
  
  <p>Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):</p>
  
  <p><a href="https://www.kaggle.com/fnguyen/titanicrising-test" rel="nofollow noreferrer">https://www.kaggle.com/fnguyen/titanicrising-test</a></p>
  
Tags: <machine-learning><prediction>
Title: Why is my model accuracy high in train-test split but actually worse than chance in validation set?
UserDisplayName: Fnguyen
ThreadText: # Why is my model accuracy high in train-test split but actually worse than chance in validation set?
  
  (Asked by: username_0 on 2020-01-31)
  
  I have trained a XGboost model to predict survival for the [Kaggle Titanic ML competition](https://www.kaggle.com/c/titanic/overview).
  
  As with all Kaggle competitions there is a `train` dataset with the target variable included and a `test` dataset without the target variable which is used by Kaggle to compute the final accuracy score that determines your leaderboard ranking.
  
  **My problem:**
  
  I have build a fairly simple ensemble classifier (based on XGboost) and evaluated it via standard train-test-splits of the `train` data. The **accuracy** I get from this **validation is ~80%** which is good but not amazing by public leaderboard standards (excluding the 100% cheaters).
  
  The results and all the KPIs I looked at of this standard model do not indicate severe overfitting, etc. to me.
  
  However when I submit my predictions for the `test` set my **public score is ~35%** which is way below even a random chance model. It is sooo bad I even improved my score by simply reversing all predictions from the model.
  
  **Why is my model so much worse on the `test`?**
  
  I know that Kaggle computes their scores a bit differently than I do locally, additionally there is probably some differences between the datasets. Most who join the competition notices at least some difference between their local test scores and the public scores.
  
  However my difference is really drastic and indeed reversing the predictions improves my score. This does not make sense to me because reversing the predictions on my local validations leads to garbage predictions, so this is not a simple problem of generally reversed predictions.
  
  **So can you help me understand how those two issues happen at the same time:**
  
  * Drastic difference between local accuracy and public score
  * Reversing actually leads to the better public score.
  
  Here is my notebook for the code (please ignore the errors, they are simply because the code does not work on kaggle kernels only locally):
  
  <https://www.kaggle.com/fnguyen/titanicrising-test>
  
  **Tags:** machine-learning, prediction
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-03. Score: 2 | Accepted answer
  
  Looking at your code, one set of data transformations were applied to the train data and a different set of transformations were applied to the test data. Different data transformations could account for different evaluation metric performance.
  
  It is best practices to put all data transformations in a function so they can be applied to all data in a similar way.
  
  Since you are using scikit-learn, `sklearn.compose.ColumnTransformer` is designed for this purpose. Example code for the Titanic dataset is [here](https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html).
  
  > **Comments:**
  > **username_0**: I will try that but functionally the data transformation for both datasets is the same with the only difference being that the test data does not contain a "Ms" at factor level. Nevertheless you never know what makes a quirk in the data so I will package all transformations in a function and apply the same to both sets to eliminate this factor.  
  > **username_0**: I have cleaned up my code using your answer and a notebook I found on Kaggle as inspiration. By joining test and train dataset first, then doing all necessary transformations and only after splitting them again for modeling and prediction I was able to remove the error. They key to his was actually that I was applying an Imputation Method, I will describe that in an answer in detail.  
  
  ---
  
  ### Answer by username_0 on 2020-02-05. Score: 0
  
  @BrianSpiering was generally correct in pointing out that you should always apply the same transformations to your train and test dataset.
  
  This was the key to my solution which was a bit more specific but might actually help others who encounter a similar problem.
  
  Specifically my mistake came about because of **Imputation**! Some of the factors I used for my model were `NA` in both the train and test data set. To complete the data I simply imputed these missings using mean and mode respectively. However since I did those transformations separately on both sets the actual mean/mode value that was used differed heavily! By applying the imputation on the full data I also imputed the same data for all missing cases which solved my error.
  
  My resulting accuracy in the public leaderboard is now at 74.2% which is fairly close to my local test score of 79.6%.
  
  
  ---
  
  