Answers.Body: <p><em><strong>we are recognizing the disease, not the person.</strong></em></p>
  <p>If you're training a computer vision model with only images and no auxiliary information then a randomized sampling should be enough to prevent the model from over fitting on x-ray scans taken on the same person.</p>
  <p><em><strong>If images from the same person exist in both subsets, the problem will be easy, and not reasonable, from a practical point of view. Do you agree?</strong></em></p>
  <p>Only partially. The aim of a test set is to allow you to quantify how well your model will perform on a real use case. Which is close but not the same as making the problem as hard as possible in testing phase.</p>
  <p>Also, it is not necessarily true that having 20 examples from the same person in the training set will lead to high accuracy on even 1 single test instance coming again from the same person. This is because those training instances might contain bias (like many tumors on the left side of the scan rather than the right side).</p>
  <p>This is why I personally would go for the following splitting approach:</p>
  <ul>
  <li>sample <strong>from the whole dataset</strong> a small group of people (for example top n people with fewest amount of scans). Let's call this set <em>between groups test set</em>.</li>
  <li>sample <strong>per person</strong> (from the remaining data) 70% training instances and 30% test instances. Let's call it <em>within groups test set</em>.</li>
  </ul>
  <p>During validation I would then perform the following checks:</p>
  <ul>
  <li>metrics calculation (precision, recall, f-score, i.e the whole package) on the between group test performed as usual.</li>
  <li>metrics calculation <strong>per person</strong> (or clusters of people, paying attention in keeping the clusters the same at every validation step) on the within groups test set.</li>
  </ul>
  <p>The second check is the most interesting. Of course there will be some random variation in the performance, but if the model will start performing better on some specific clusters of people that might be a hint that the model is learning some confounding variables probably related to the presence of really similar scans or other bias. testing only on the between groups will not let you to draw that conclusion.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-11T18:10:54.153Z
Answers.LastActivityDate: 2021-11-11T18:10:54.153Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have a database that contains healthy persons and lung cancer patients. I need to design a deep neural network for the binary classification problem (cancer/no cancer). I need to split the dataset into 70% train and 30% test.</p>
  <p>How can I do the splitting? According to persons?</p>
  <p>I think that splitting according to persons is correct since this will ensure that the same person will not exist simultaneously in both the training and the test subsets. This is reasonable since we are recognizing the disease, not the person. If images from the same person exist in both subsets, the problem will be easy, and not reasonable, from a practical point of view. Do you agree?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T11:26:45.607Z
Comments.Text: it is not clear to me what other alternatives you have. From the way you describe the data it seems you have information per each person, e.g. [age:25, weight:60, height: 1.80, blood_pressure:120, ..., cancer: False]. By splitting into train test, since you have one record per person, it would be impossible to have the same person into train and test sets at the same time. Am I missing something?
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-10T19:55:05.503Z
Comments.Text: I have many samples for the same person (10 samples for examples)). The samples represent X rays images from the person.
Comments.UserDisplayName: Noha
Tags: <deep-learning><binary-classification><cross-validation><training-datasets><test-datasets>
Title: Given a dataset of people with and without cancer, should I split it into training and test datasets such that the same person is not in both?
UserDisplayName: Noha
ThreadText: # Given a dataset of people with and without cancer, should I split it into training and test datasets such that the same person is not in both?
  
  (Asked by: username_0 on 2021-11-09)
  
  I have a database that contains healthy persons and lung cancer patients. I need to design a deep neural network for the binary classification problem (cancer/no cancer). I need to split the dataset into 70% train and 30% test.
  
  How can I do the splitting? According to persons?
  
  I think that splitting according to persons is correct since this will ensure that the same person will not exist simultaneously in both the training and the test subsets. This is reasonable since we are recognizing the disease, not the person. If images from the same person exist in both subsets, the problem will be easy, and not reasonable, from a practical point of view. Do you agree?
  
  **Tags:** deep-learning, binary-classification, cross-validation, training-datasets, test-datasets
  
  > **Comments:**
  > **username_1**: it is not clear to me what other alternatives you have. From the way you describe the data it seems you have information per each person, e.g. [age:25, weight:60, height: 1.80, blood_pressure:120, ..., cancer: False]. By splitting into train test, since you have one record per person, it would be impossible to have the same person into train and test sets at the same time. Am I missing something?  
  > **username_0**: I have many samples for the same person (10 samples for examples)). The samples represent X rays images from the person.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-11. Score: 1
  
  ***we are recognizing the disease, not the person.***
  
  If you're training a computer vision model with only images and no auxiliary information then a randomized sampling should be enough to prevent the model from over fitting on x-ray scans taken on the same person.
  
  ***If images from the same person exist in both subsets, the problem will be easy, and not reasonable, from a practical point of view. Do you agree?***
  
  Only partially. The aim of a test set is to allow you to quantify how well your model will perform on a real use case. Which is close but not the same as making the problem as hard as possible in testing phase.
  
  Also, it is not necessarily true that having 20 examples from the same person in the training set will lead to high accuracy on even 1 single test instance coming again from the same person. This is because those training instances might contain bias (like many tumors on the left side of the scan rather than the right side).
  
  This is why I personally would go for the following splitting approach:
  
  * sample **from the whole dataset** a small group of people (for example top n people with fewest amount of scans). Let's call this set *between groups test set*.
  * sample **per person** (from the remaining data) 70% training instances and 30% test instances. Let's call it *within groups test set*.
  
  During validation I would then perform the following checks:
  
  * metrics calculation (precision, recall, f-score, i.e the whole package) on the between group test performed as usual.
  * metrics calculation **per person** (or clusters of people, paying attention in keeping the clusters the same at every validation step) on the within groups test set.
  
  The second check is the most interesting. Of course there will be some random variation in the performance, but if the model will start performing better on some specific clusters of people that might be a hint that the model is learning some confounding variables probably related to the presence of really similar scans or other bias. testing only on the between groups will not let you to draw that conclusion.
  
  
  ---
  
  