Body: <p>I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I have around 2.5 million samples, splitted 8/1/1 for train/test/validation. I am seeing a uniform gap between training and validation results on various metrics. E.g. see the graphs below. What are the potential explanations for such phenomenon? My conjecture would be some sort of information leakage issue, but I double checked the data and re-did train/test/validation split and the gaps persisted.</p>
  <p>EDIT: I am using <code>train_test_split</code> from <code>sklearn</code>:</p>
  <pre><code>df_train, df_test = train_test_split(
      df, test_size=0.2,
  )
  
  df_test, df_val = train_test_split(
      df_test, test_size=0.5,
  )
  
  </code></pre>
  <p><a href="https://i.sstatic.net/GqEnj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GqEnj.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/0UM1D.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0UM1D.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/qbsOM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qbsOM.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-01T16:11:20.940Z
Comments.Text: are you 100% sure you're splitting properly? Did you check that none of the filenames of your training set are present in the test/validation ones? You consistently have higher scores in validation and lower loss, which is a huge hint supporting you're sampling a subset for test and validation without actually removing the instances from the training set.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-01T16:28:26.167Z
Comments.Text: @EdoardoGuerriero I am simply using sklearn's train_test_split on my dataframe to do the job, and my df does not contain any duplicate. Added a code snippet on the splitting part
Comments.UserDisplayName: Ryan
Tags: <deep-learning><loss><cross-validation><metric>
Title: uniform gap between training and validation metrics
UserDisplayName: Ryan
ThreadText: # uniform gap between training and validation metrics
  
  (Asked by: username_0 on 2022-08-01)
  
  I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I have around 2.5 million samples, splitted 8/1/1 for train/test/validation. I am seeing a uniform gap between training and validation results on various metrics. E.g. see the graphs below. What are the potential explanations for such phenomenon? My conjecture would be some sort of information leakage issue, but I double checked the data and re-did train/test/validation split and the gaps persisted.
  
  EDIT: I am using `train_test_split` from `sklearn`:
  
  ```
  df_train, df_test = train_test_split(
      df, test_size=0.2,
  )
  
  df_test, df_val = train_test_split(
      df_test, test_size=0.5,
  )
  
  ```
  
  [![enter image description here](https://i.sstatic.net/GqEnj.png)](https://i.sstatic.net/GqEnj.png)
  [![enter image description here](https://i.sstatic.net/0UM1D.png)](https://i.sstatic.net/0UM1D.png)
  [![enter image description here](https://i.sstatic.net/qbsOM.png)](https://i.sstatic.net/qbsOM.png)
  
  **Tags:** deep-learning, loss, cross-validation, metric
  
  > **Comments:**
  > **username_1**: are you 100% sure you're splitting properly? Did you check that none of the filenames of your training set are present in the test/validation ones? You consistently have higher scores in validation and lower loss, which is a huge hint supporting you're sampling a subset for test and validation without actually removing the instances from the training set.  
  > **username_0**: @username_1 I am simply using sklearn's train_test_split on my dataframe to do the job, and my df does not contain any duplicate. Added a code snippet on the splitting part  
  
  ---
  
  ## 0 answers
  
  