Answers.Body: <p>Wow, I would just go for accuarcy as the singular most important measure, but be careful of sampling bias. </p>
  
  <p>If the classification training had an unbiased sample* and you have an accuracy of >90%, in general ANN that wouldn't be great, but in bioinformatcs thats really good. If your training comprises sampling bias that gets tricky and you either need to accomodate that in your accuarcy ROC etc ... or else retrain to eliminate the sample bias.</p>
  
  <p>You can account for all of them of course by wrapping everything into a single analysis.</p>
  
  <p>*, by this I mean that each of the 3 classifications comprises 33% in each training set. If this is badly skewed, this requires consideration and gets complicated. If it is precisely 1/3 each then 'accurary' is the key stat.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T19:45:56.343Z
Answers.Text: thank you!how can i calculate accuracy for all the three categories? is it correct to calculate the mean ? (sorry in advance for this question)
Answers.UserDisplayName: marilu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T19:51:25.717Z
Answers.Text: Thanks, firstly this looks very cool work, again... its cool.  Did you do 3 independent training sets, each with a single classification OR a single training with 3 classifications?
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T20:35:23.293Z
Answers.Text: a single training with 3 classifications.thank you!
Answers.UserDisplayName: marilu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T21:23:40.933Z
Answers.Text: I get it, sorry. Yes and no. Yes: for your own internal purpose yes and a line or two in the report/paper. You don't want major skews between the classifications (these are different skews from the sampling skews). No: for the overall accuracy score, its one overall number (obviously one which isn't hiding a skew). Essentially the answer is its just one number, but you need to be confident its representative. I hope thats ok.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T21:31:56.820Z
Answers.Text: If you get 90% accuracy with a 33% sample for training of each classification, then its taken as read that the misclassification has reasonable distribution. Leading bioinformatics publications have present been as low as 80%.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T06:28:12.503Z
Answers.Text: so to sum up, instead of  making a 3*3 matrix to measure accuracy, I could calculate the mean (because now I have constructed matrices for each category). and if this is higher than 90% (but even less) then my model is good. am I correct? Thank you so much!!!
Answers.UserDisplayName: marilu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T08:07:13.490Z
Answers.Text: Basically yes in my opinion 90% would be fantastic and you are sorted, look at past papers (Bioinformatics journal, to verify the accuarcy against their work ...  Your confusion matrix (the formal name for a 3*3 matrix) helps diagnose your training, if your accuracy is low, scikit-learn has an automatic function for this. Personally I think 85%+ is good, but just my opinion.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T09:38:17.007Z
Answers.Text: thank you!in order to select one architecture than another how much should be the difference in accuracy? for example from 0.856 to 0.860 is it considered imortant difference?from 0.85 to 0.86?or it needs to be even bigger lets say from 0.85 to 0.90 or more?thanks!!
Answers.UserDisplayName: marilu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T11:16:56.660Z
Answers.Text: Either 0.856 or 0.86, sounds like what you really have, I think is fine and shows archecture convergence. Please refer to other comparable papers however for accuracy comparision. Obviously 90% is better. Things get very complicated indeed increasing the accuracy.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T16:17:12.603Z
Answers.LastActivityDate: 2020-05-01T18:44:28.037Z
Answers.LastEditDate: 2020-05-01T18:44:28.037Z
Answers.UserDisplayName: M__
Body: <p>I have made some architectures of neural networks in order to classify proteins in three categories. I have calculated true positive, true negative, false negative, false positive, Mathews, sensitivity, specificity, accuracy for each category. My question is what is the right way to evaluate the architecture. Except for the c-index, could I calculate a cumulative sensitivity, accuracy etc?or should I evaluate separately for each category? Thank you!</p>
  
Tags: <proteins><networks>
Title: how to evaluate neural networks
ThreadText: # how to evaluate neural networks
  
  (Asked by: username_0 on 2020-05-01)
  
  I have made some architectures of neural networks in order to classify proteins in three categories. I have calculated true positive, true negative, false negative, false positive, Mathews, sensitivity, specificity, accuracy for each category. My question is what is the right way to evaluate the architecture. Except for the c-index, could I calculate a cumulative sensitivity, accuracy etc?or should I evaluate separately for each category? Thank you!
  
  **Tags:** proteins, networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-01. Score: 0 | Accepted answer
  
  Wow, I would just go for accuarcy as the singular most important measure, but be careful of sampling bias.
  
  If the classification training had an unbiased sample\* and you have an accuracy of >90%, in general ANN that wouldn't be great, but in bioinformatcs thats really good. If your training comprises sampling bias that gets tricky and you either need to accomodate that in your accuarcy ROC etc ... or else retrain to eliminate the sample bias.
  
  You can account for all of them of course by wrapping everything into a single analysis.
  
  \*, by this I mean that each of the 3 classifications comprises 33% in each training set. If this is badly skewed, this requires consideration and gets complicated. If it is precisely 1/3 each then 'accurary' is the key stat.
  
  > **Comments:**
  > **username_0**: thank you!how can i calculate accuracy for all the three categories? is it correct to calculate the mean ? (sorry in advance for this question)  
  > **username_1**: Thanks, firstly this looks very cool work, again... its cool.  Did you do 3 independent training sets, each with a single classification OR a single training with 3 classifications?  
  > **username_0**: a single training with 3 classifications.thank you!  
  > **username_1**: I get it, sorry. Yes and no. Yes: for your own internal purpose yes and a line or two in the report/paper. You don't want major skews between the classifications (these are different skews from the sampling skews). No: for the overall accuracy score, its one overall number (obviously one which isn't hiding a skew). Essentially the answer is its just one number, but you need to be confident its representative. I hope thats ok.  
  > **username_1**: If you get 90% accuracy with a 33% sample for training of each classification, then its taken as read that the misclassification has reasonable distribution. Leading bioinformatics publications have present been as low as 80%.  
  > **username_0**: so to sum up, instead of  making a 3*3 matrix to measure accuracy, I could calculate the mean (because now I have constructed matrices for each category). and if this is higher than 90% (but even less) then my model is good. am I correct? Thank you so much!!!  
  > **username_1**: Basically yes in my opinion 90% would be fantastic and you are sorted, look at past papers (Bioinformatics journal, to verify the accuarcy against their work ...  Your confusion matrix (the formal name for a 3*3 matrix) helps diagnose your training, if your accuracy is low, scikit-learn has an automatic function for this. Personally I think 85%+ is good, but just my opinion.  
  > **username_0**: thank you!in order to select one architecture than another how much should be the difference in accuracy? for example from 0.856 to 0.860 is it considered imortant difference?from 0.85 to 0.86?or it needs to be even bigger lets say from 0.85 to 0.90 or more?thanks!!  
  > **username_1**: Either 0.856 or 0.86, sounds like what you really have, I think is fine and shows archecture convergence. Please refer to other comparable papers however for accuracy comparision. Obviously 90% is better. Things get very complicated indeed increasing the accuracy.  
  
  ---
  
  