Answers.Body: <p>You are missing a validation split triggering data leakage. The error is here:</p>
  <pre><code>grid_search.fit(X_train,y_train)
  </code></pre>
  <p>You can't use <code>X_train</code> and <code>y_train</code> here. The correction is</p>
  <pre><code>grid_search.fit(X_val,y_val)
  </code></pre>
  <p>Generating <code>X_val</code> and <code>y_val</code> is described below.</p>
  <p>The code used to generate</p>
  <blockquote>
  <p>Training Accuracy: 0.967032967032967
  Test Accuracy: 0.782608695652174</p>
  </blockquote>
  <p>...  included one <code>test_split_train</code> and you need to perform a second <code>test_split_train</code> to obtain <code>X_val</code>, and <code>y_val</code>. The original code gave a pretty good as a result, the training score will always be higher.</p>
  <p>The diagnosis of the code presented is there is 100% data leakage. What is missing is a second <code>test_train_split</code>. Again one <code>test_train_split</code> is present in the original code and there would have been a,</p>
  <pre><code>import pandas as pd
  from sklearn.model_selection import train_test_split
  
  X = # a dataframe containing the features remove the target y (below)
  y = # a same dataframe but a single column which is the target
  X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=1)
  </code></pre>
  <p>The above will give a training proportion at 80% and test proportion at 20%. It's called an 80:20 split. That code will have been in the original code for the first calculation, I know that because the results are pretty standard for a routine ML <code>test_train_split</code> operation. Whats happening is the training set is being held absolutely separately to the test set. There is 100% compartmentalisation between the two data sets. Thats critical because if they got confused the calculation falls over, you will be testing the exact data thats just been trained. That confusion would be term &quot;data leakage&quot;.</p>
  <p>In the code you presented what has happened is the same data used for parameterisation is exactly the same data for training (<code>X_train</code>, <code>y_train</code>). Again it's called data leakage and parameter grids are very sensitive to this.</p>
  <p>To correct this when a parameter grid is used it must have its own data set. That is instead of having a <code>X_train,X_test,y_train,y_test</code> the <code>test_train_split</code> is modified to generate <code>X_train,X_test,y_train,y_test, X_val, y_val</code>. The code to generate this is,</p>
  <pre><code>import pandas as pd
  from sklearn.model_selection import train_test_split
   X_train, X_test, y_train, y_test 
      = train_test_split(X, y, test_size=0.2, random_state=1)
  
   X_train, X_val, y_train, y_val 
      = train_test_split(X_train, y_train, test_size=0.25, random_state=1)
  </code></pre>
  <p>This approach is singly leveraging <code>test_train_split</code> by splitting the training data set twice. This example creates a 60:20:20 split. Some say that 70:20:10 split is used for a parameter split (10 for the parameter search). Matter of opinion (you just adjust the proportions).</p>
  <p><strong>Thus</strong></p>
  <ol>
  <li>the grid search is strictly for the parameter split (X_val, y_val)</li>
  <li>and those parameters at then using in training split (X_train, y_train)</li>
  <li>the testing is then performed (X_test, y_test)</li>
  </ol>
  <p>What this process does is ensure no part of <code>X_val</code>, <code>y_val</code> is ever seen in the training set. If that happens again - data leakage. Here only the optimised parameters are seen by the training set.</p>
  <hr />
  <p><strong>Summary</strong> Introducing validation parameter split into the calculation isn't trivial and there is nothing wrong with your original result. Thus you don't necessarily need it. Generally speaking 80% accuracy is fine, yours is 1% lower (can be overlooked).</p>
  <p>Parameter tuning will increase the accuracy score but it will be 85% rather than 79% and if it's really gone well then it might be 90%. What I'm saying is it will improve things, but will not absolutely solve the accuracy issue, it's just a general and accepted issue within ML.</p>
  <hr />
  <p>Just to finally add there are biological reasons in cancer prediction why accuracy is a key parameter, but thats a different question.</p>
  <hr />
  <p>Comments: causes lower scores</p>
  <p>What I'm saying the correct approach. I recognise that <a href="https://www.geeksforgeeks.org/svm-hyperparameter-tuning-using-gridsearchcv-ml/" rel="nofollow noreferrer"><strong>Geek for Geeks</strong></a> uses your approach - but they are wrong.</p>
  <p>A separate validation split will certainly give result in lower scores than <code>grid_search.fit(X_train,y_train)</code> thats inevitable, but it is correct in stopping overfitting.</p>
  <p>If the validation split causes its lower accuracy than <code>0.8</code> I'd look at <code>gammas = ... 0.0001,0.001,0.1 ...</code> because you have missed <code>0.01</code> and that will affect your model. It shouldn't give a lower score than the default parameters - it's not possible and something is wrong with the code and  application of parameters.</p>
  <p>Once you've the initial parameters you can refine the parameter search around the initial values - nothing wrong with that and that can even be coded (automated). The only minor issue is reducing the size of the training split (60% is a bit low). That can cause issues in which case the split is 70:20:10, i.e. 10 for validation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T20:38:53.940Z
Answers.Text: Thanks for your reply, but I tried what you suggested but it gave me lower accuracy values.
Answers.UserDisplayName: user17818805
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T20:50:56.547Z
Answers.Text: @user17818805 answered above below the last -------
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T20:56:55.793Z
Answers.Text: I understood you the least lost data will infer my model. I can't improve my accuracy, I've changed the model inputs leaving only the OTUs, I've tried the PCA but to no avail.
Answers.UserDisplayName: user17818805
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T21:49:00.893Z
Answers.Text: I did'nt get that 'Once you've the initial parameters you can refine the parameter search around the initial values - nothing wrong with that and that can even be coded (automated). The only minor issue is reducing the size of the training split (60% is a bit low). That can cause issues in which case the split is 70:20:10, i.e. 10 for validation'.
Answers.UserDisplayName: user17818805
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-25T10:34:32.957Z
Answers.LastActivityDate: 2023-08-28T21:11:11.087Z
Answers.LastEditDate: 2023-08-28T21:11:11.087Z
Answers.UserDisplayName: M__
Body: <p>I'm working on an ML model to predict colorectal cancer using data from the human microbiome (collected from fecal samples).</p>
  <p>My dataset is made up of metadata (containing phenotypic information on the N=156 population on which the study is being carried out,
  <a href="https://i.sstatic.net/MkOVB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MkOVB.png" alt="Metadata features" /></a> and an OTU table <a href="https://i.sstatic.net/AArhv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AArhv.png" alt="OTU table have 129 column containing SUBJECT IDs and the 1st column contain taxa" /></a>containing a taxonomy of bacteria found and their abundance for each sample from one person (each person has one and only one sample).</p>
  <p>I filtered the OTU table by eliminating sparse data (I eliminated OTUs below a certain threshold) after which I calculated the relative abundance for each OTU.<br />
  I deleted samples with missing values.<br />
  I made the metadata join (Subject ID    Sample ID    Age (years)    Gender    BMI (kg/m²)    Country of Residence    Diagnosis    TNM Stage    AJCC Stage    Localization) and the OTU table.</p>
  <p>In the end, I ended up with 114 rows × 221 columns .</p>
  <p>I normalized the data and then built an SVM(SVC) model with the rbf kernel with C=0.05 and gamma=0.001 I got</p>
  <pre><code>Training Accuracy: 0.967032967032967
  Test Accuracy: 0.782608695652174
  SVC(AUC=0.64)
  </code></pre>
  <p>When I try to optimize the model using the code below, the values I get for gamma and C give me overfitting (trainset accuracy is 100%).</p>
  <pre class="lang-py prettyprint-override"><code># import packages
  from sklearn.svm import SVC
  from sklearn.model_selection import GridSearchCV
  # range of values to try for gamma and C
  Cs = [0.00001,0.0001,0.001,0.1,1,2,5,10,100,1000]
  gammas = [0.00001,0.0001,0.001,0.1,1,10,100,10000] 
  # number of segments for cross validation
  nfolds = 5 # number of segments for class validation
  param_grid = {'C':Cs ,'gamma': gammas}
  # search for the best parameters using cross-validation 
  grid_search = GridSearchCV(SVC(kernel='rbf'),param_grid,cv=nfolds)
  grid_search.fit(X_train,y_train)
  grid_search.best_params_
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T06:05:15.320Z
Comments.Text: It seems like an overfitting problem, a sample of the dataset would be appreciated, I think the model might be hanging onto a few features to make the prediction.
Comments.UserDisplayName: Neeleshwar Pandey
Tags: <machine-learning><modelling><microbial-genomics><microbiome>
Title: Model optimization causes overfitting
ThreadText: # Model optimization causes overfitting
  
  (Asked by: username_0 on 2023-08-23)
  
  I'm working on an ML model to predict colorectal cancer using data from the human microbiome (collected from fecal samples).
  
  My dataset is made up of metadata (containing phenotypic information on the N=156 population on which the study is being carried out,
  [![Metadata features](https://i.sstatic.net/MkOVB.png)](https://i.sstatic.net/MkOVB.png) and an OTU table [![OTU table have 129 column containing SUBJECT IDs and the 1st column contain taxa](https://i.sstatic.net/AArhv.png)](https://i.sstatic.net/AArhv.png)containing a taxonomy of bacteria found and their abundance for each sample from one person (each person has one and only one sample).
  
  I filtered the OTU table by eliminating sparse data (I eliminated OTUs below a certain threshold) after which I calculated the relative abundance for each OTU.  
  I deleted samples with missing values.  
  I made the metadata join (Subject ID Sample ID Age (years) Gender BMI (kg/m²) Country of Residence Diagnosis TNM Stage AJCC Stage Localization) and the OTU table.
  
  In the end, I ended up with 114 rows × 221 columns .
  
  I normalized the data and then built an SVM(SVC) model with the rbf kernel with C=0.05 and gamma=0.001 I got
  
  ```
  Training Accuracy: 0.967032967032967
  Test Accuracy: 0.782608695652174
  SVC(AUC=0.64)
  
  ```
  
  When I try to optimize the model using the code below, the values I get for gamma and C give me overfitting (trainset accuracy is 100%).
  
  ```py
  # import packages
  from sklearn.svm import SVC
  from sklearn.model_selection import GridSearchCV
  # range of values to try for gamma and C
  Cs = [0.00001,0.0001,0.001,0.1,1,2,5,10,100,1000]
  gammas = [0.00001,0.0001,0.001,0.1,1,10,100,10000] 
  # number of segments for cross validation
  nfolds = 5 # number of segments for class validation
  param_grid = {'C':Cs ,'gamma': gammas}
  # search for the best parameters using cross-validation 
  grid_search = GridSearchCV(SVC(kernel='rbf'),param_grid,cv=nfolds)
  grid_search.fit(X_train,y_train)
  grid_search.best_params_
  
  ```
  
  **Tags:** machine-learning, modelling, microbial-genomics, microbiome
  
  > **Comments:**
  > **username_1**: It seems like an overfitting problem, a sample of the dataset would be appreciated, I think the model might be hanging onto a few features to make the prediction.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-25. Score: 1
  
  You are missing a validation split triggering data leakage. The error is here:
  
  ```
  grid_search.fit(X_train,y_train)
  
  ```
  
  You can't use `X_train` and `y_train` here. The correction is
  
  ```
  grid_search.fit(X_val,y_val)
  
  ```
  
  Generating `X_val` and `y_val` is described below.
  
  The code used to generate
  
  > Training Accuracy: 0.967032967032967
  > Test Accuracy: 0.782608695652174
  
  ... included one `test_split_train` and you need to perform a second `test_split_train` to obtain `X_val`, and `y_val`. The original code gave a pretty good as a result, the training score will always be higher.
  
  The diagnosis of the code presented is there is 100% data leakage. What is missing is a second `test_train_split`. Again one `test_train_split` is present in the original code and there would have been a,
  
  ```
  import pandas as pd
  from sklearn.model_selection import train_test_split
  
  X = # a dataframe containing the features remove the target y (below)
  y = # a same dataframe but a single column which is the target
  X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,random_state=1)
  
  ```
  
  The above will give a training proportion at 80% and test proportion at 20%. It's called an 80:20 split. That code will have been in the original code for the first calculation, I know that because the results are pretty standard for a routine ML `test_train_split` operation. Whats happening is the training set is being held absolutely separately to the test set. There is 100% compartmentalisation between the two data sets. Thats critical because if they got confused the calculation falls over, you will be testing the exact data thats just been trained. That confusion would be term "data leakage".
  
  In the code you presented what has happened is the same data used for parameterisation is exactly the same data for training (`X_train`, `y_train`). Again it's called data leakage and parameter grids are very sensitive to this.
  
  To correct this when a parameter grid is used it must have its own data set. That is instead of having a `X_train,X_test,y_train,y_test` the `test_train_split` is modified to generate `X_train,X_test,y_train,y_test, X_val, y_val`. The code to generate this is,
  
  ```
  import pandas as pd
  from sklearn.model_selection import train_test_split
   X_train, X_test, y_train, y_test 
      = train_test_split(X, y, test_size=0.2, random_state=1)
  
   X_train, X_val, y_train, y_val 
      = train_test_split(X_train, y_train, test_size=0.25, random_state=1)
  
  ```
  
  This approach is singly leveraging `test_train_split` by splitting the training data set twice. This example creates a 60:20:20 split. Some say that 70:20:10 split is used for a parameter split (10 for the parameter search). Matter of opinion (you just adjust the proportions).
  
  **Thus**
  
  1. the grid search is strictly for the parameter split (X\_val, y\_val)
  2. and those parameters at then using in training split (X\_train, y\_train)
  3. the testing is then performed (X\_test, y\_test)
  
  What this process does is ensure no part of `X_val`, `y_val` is ever seen in the training set. If that happens again - data leakage. Here only the optimised parameters are seen by the training set.
  
  ---
  
  **Summary** Introducing validation parameter split into the calculation isn't trivial and there is nothing wrong with your original result. Thus you don't necessarily need it. Generally speaking 80% accuracy is fine, yours is 1% lower (can be overlooked).
  
  Parameter tuning will increase the accuracy score but it will be 85% rather than 79% and if it's really gone well then it might be 90%. What I'm saying is it will improve things, but will not absolutely solve the accuracy issue, it's just a general and accepted issue within ML.
  
  ---
  
  Just to finally add there are biological reasons in cancer prediction why accuracy is a key parameter, but thats a different question.
  
  ---
  
  Comments: causes lower scores
  
  What I'm saying the correct approach. I recognise that [**Geek for Geeks**](https://www.geeksforgeeks.org/svm-hyperparameter-tuning-using-gridsearchcv-ml/) uses your approach - but they are wrong.
  
  A separate validation split will certainly give result in lower scores than `grid_search.fit(X_train,y_train)` thats inevitable, but it is correct in stopping overfitting.
  
  If the validation split causes its lower accuracy than `0.8` I'd look at `gammas = ... 0.0001,0.001,0.1 ...` because you have missed `0.01` and that will affect your model. It shouldn't give a lower score than the default parameters - it's not possible and something is wrong with the code and application of parameters.
  
  Once you've the initial parameters you can refine the parameter search around the initial values - nothing wrong with that and that can even be coded (automated). The only minor issue is reducing the size of the training split (60% is a bit low). That can cause issues in which case the split is 70:20:10, i.e. 10 for validation.
  
  > **Comments:**
  > **username_0**: Thanks for your reply, but I tried what you suggested but it gave me lower accuracy values.  
  > **username_2**: @username_0 answered above below the last -------  
  > **username_0**: I understood you the least lost data will infer my model. I can't improve my accuracy, I've changed the model inputs leaving only the OTUs, I've tried the PCA but to no avail.  
  > **username_0**: I did'nt get that 'Once you've the initial parameters you can refine the parameter search around the initial values - nothing wrong with that and that can even be coded (automated). The only minor issue is reducing the size of the training split (60% is a bit low). That can cause issues in which case the split is 70:20:10, i.e. 10 for validation'.  
  
  ---
  
  