Answers.Body: <p>You could be seeing the effect of SNPs on haplotype-driven conditions (i.e. where a combination of multiple variants predicts the class, rather than a single variant). </p>
  
  <p>Do you have any method to determine the age of particular variants (e.g. via an LD plot to find inconsistencies between variants and the surrounding block structure), so that you can see if that correlates with the likelihood of error?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T11:42:54.957Z
Answers.Text: No, unforetunately no.
Answers.UserDisplayName: Giannis Lazaridis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-12T08:24:51.463Z
Answers.LastActivityDate: 2020-04-12T08:24:51.463Z
Answers.UserDisplayName: gringer
Body: <p>First of all, I'd like to apologize for any spelling or grammar mistakes.</p>
  
  <p>I'm having a problem using R for a classification problem. My dataset contains ~300.000 genomic data, and the features are DNA-related features (number of dinucleotides, number of trinucleotides, the CG Content, and some more). In conclusion, I have a dataset of 300.000 rows and 84 columns (columns= features). The 84th feature is basically the classification variable (there are two classes: class 1 and class 2).</p>
  
  <p>I split the dataset in train set and test set and fit the train set to create a model using different methods, like Random Forest, XGBoost, Adaboost, etc. and check the accuracy using the test set.</p>
  
  <p>In every model, the accuracy of the prediction on the test set is always around 86-87%. What I noticed is that, in the test set, for EVERY MODEL, almost the same number of rows (data) is predicted right (the model predict its true class), and always almost the same data of rows (data) is predicted wrong (the model predict wrong class. For example its true class is 1 but the model predicts 2).</p>
  
  <p>If I take the set of data that are predicted right for every model I've created (250.000 data from the 300.000), and build a new model out of them (splitting them again in a new train and test set), I have an accuracy of 98%.</p>
  
  <p>That means that the other 50.000 data (from the 300.000), add an error at every model I'm trying to create.</p>
  
  <p>But I can't just ignore them. I thought that it would be a way to create two differents models, but doesn't know how to figure this out.</p>
  
  <p>Then I thought that there could be a way to find which variables from the two subsets have strong differences, but even if I figure this out, I don't know what's the way for example to scale the original dataset to prevent those differences from keeping the accuracy in 87% in every model (using the original dataset of the 300.000 rows).</p>
  
  <p>In conclusion, I have a dataset that pretends like "two datasets". It consists of two subsets: the first one is always predicted right from any model (machine learning algorithm) I'm trying, and the second one is always predicted wrong. How could I build a strong model taking into account the differences?</p>
  
  <p>I also took a look at the box plots of the features. The most important feature for every model is called "CG" (number of dinucleotide of cytosines and guanines in DNA). In the following image you can see the boxplot of CG feature separating the original dataset in two datasets (T = for the dataset that is predicted right in every model) and (F = for the dataset that is predicted wrong in every model). 1 stands for data of class 1 and 2 stands for data of class 2. So, T1 means data from the right predicted dataset of class 1, T2 means data from the right predicted dataset of classes 2, T means data from the right predicted dataset for both class 1 and 2, etc.</p>
  
  <p><a href="https://i.sstatic.net/ZmeEN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZmeEN.png" alt="box plot CG"></a></p>
  
  <p>As I'm a new beginner in R and in machine learning, I hope you do understand my problem. Thanks sincerely.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-09T10:35:16.793Z
Comments.Text: Are your classes balanced, meaning that do you have roughly the same number of the two classes? Looks like your models are learning a particular set of samples very well at the expense of learning others. Cross-validation could help here.
Comments.UserDisplayName: haci
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-09T13:56:15.780Z
Comments.Text: Not sure it is related but do you shuffle your datasets before training ? I had weird patterns of valid accuracy from epoch to epoch when training a neural network once, it was because I forgot to shuffle my datasets.
Comments.UserDisplayName: Paul Endymion
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-12T18:30:45.890Z
Comments.Text: It needs some thought. Whats your training/test split, 80:20? Any parameter splits?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-12T18:44:27.650Z
Comments.Text: The persistent 87-88% accuracy between models is a bit weird, but its an interesting problem
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-12T20:04:56.563Z
Comments.Text: I always used a train set with same amount of class 1 and class 2. for example, a train set of 50.000 data (25.000 of class 1 and 25.000 of class 2).
Comments.UserDisplayName: Giannis Lazaridis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-12T21:31:33.990Z
Comments.Text: Okay thats the classificaiton (I don't know what "Class I" and "Class 2" represent, but I get the idea in ML, whatever "Class" is, is the thing you are looking at. The question was the split with testing, or parameterisation. The reliability of the accuracy score across very different models suggests something is not right, but I honestly don't know what.
Comments.UserDisplayName: M__
Tags: <r><genomics><phylogenetics><machine-learning>
Title: Problem with classification model of genomic data: every machine learing model predicts wrongly almost always the same subset of dataset
ThreadText: # Problem with classification model of genomic data: every machine learing model predicts wrongly almost always the same subset of dataset
  
  (Asked by: username_0 on 2020-04-12)
  
  First of all, I'd like to apologize for any spelling or grammar mistakes.
  
  I'm having a problem using R for a classification problem. My dataset contains ~300.000 genomic data, and the features are DNA-related features (number of dinucleotides, number of trinucleotides, the CG Content, and some more). In conclusion, I have a dataset of 300.000 rows and 84 columns (columns= features). The 84th feature is basically the classification variable (there are two classes: class 1 and class 2).
  
  I split the dataset in train set and test set and fit the train set to create a model using different methods, like Random Forest, XGBoost, Adaboost, etc. and check the accuracy using the test set.
  
  In every model, the accuracy of the prediction on the test set is always around 86-87%. What I noticed is that, in the test set, for EVERY MODEL, almost the same number of rows (data) is predicted right (the model predict its true class), and always almost the same data of rows (data) is predicted wrong (the model predict wrong class. For example its true class is 1 but the model predicts 2).
  
  If I take the set of data that are predicted right for every model I've created (250.000 data from the 300.000), and build a new model out of them (splitting them again in a new train and test set), I have an accuracy of 98%.
  
  That means that the other 50.000 data (from the 300.000), add an error at every model I'm trying to create.
  
  But I can't just ignore them. I thought that it would be a way to create two differents models, but doesn't know how to figure this out.
  
  Then I thought that there could be a way to find which variables from the two subsets have strong differences, but even if I figure this out, I don't know what's the way for example to scale the original dataset to prevent those differences from keeping the accuracy in 87% in every model (using the original dataset of the 300.000 rows).
  
  In conclusion, I have a dataset that pretends like "two datasets". It consists of two subsets: the first one is always predicted right from any model (machine learning algorithm) I'm trying, and the second one is always predicted wrong. How could I build a strong model taking into account the differences?
  
  I also took a look at the box plots of the features. The most important feature for every model is called "CG" (number of dinucleotide of cytosines and guanines in DNA). In the following image you can see the boxplot of CG feature separating the original dataset in two datasets (T = for the dataset that is predicted right in every model) and (F = for the dataset that is predicted wrong in every model). 1 stands for data of class 1 and 2 stands for data of class 2. So, T1 means data from the right predicted dataset of class 1, T2 means data from the right predicted dataset of classes 2, T means data from the right predicted dataset for both class 1 and 2, etc.
  
  [![box plot CG](https://i.sstatic.net/ZmeEN.png)](https://i.sstatic.net/ZmeEN.png)
  
  As I'm a new beginner in R and in machine learning, I hope you do understand my problem. Thanks sincerely.
  
  **Tags:** r, genomics, phylogenetics, machine-learning
  
  > **Comments:**
  > **username_3**: It needs some thought. Whats your training/test split, 80:20? Any parameter splits?  
  > **username_3**: The persistent 87-88% accuracy between models is a bit weird, but its an interesting problem  
  > **username_0**: I always used a train set with same amount of class 1 and class 2. for example, a train set of 50.000 data (25.000 of class 1 and 25.000 of class 2).  
  > **username_3**: Okay thats the classificaiton (I don't know what "Class I" and "Class 2" represent, but I get the idea in ML, whatever "Class" is, is the thing you are looking at. The question was the split with testing, or parameterisation. The reliability of the accuracy score across very different models suggests something is not right, but I honestly don't know what.  
  > **username_1**: Are your classes balanced, meaning that do you have roughly the same number of the two classes? Looks like your models are learning a particular set of samples very well at the expense of learning others. Cross-validation could help here.  
  > **username_2**: Not sure it is related but do you shuffle your datasets before training ? I had weird patterns of valid accuracy from epoch to epoch when training a neural network once, it was because I forgot to shuffle my datasets.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2020-04-12. Score: 0
  
  You could be seeing the effect of SNPs on haplotype-driven conditions (i.e. where a combination of multiple variants predicts the class, rather than a single variant).
  
  Do you have any method to determine the age of particular variants (e.g. via an LD plot to find inconsistencies between variants and the surrounding block structure), so that you can see if that correlates with the likelihood of error?
  
  > **Comments:**
  > **username_0**: No, unforetunately no.  
  
  ---
  
  