Answers.Body: <p>Accuracy can sometimes be a very coarse metric. When it is applied to three class problems, people often take the class label with maximum predicted probability and predict that. The probabilities of the individual labels are ignored. I'd recommend that as well as accuracy you calculate sensitivity and specificity for each class and the area under the ROC curve. For both of these, you can take a 1 vs rest (i.e. class 1 vs classes 2 &amp; 3, class 2 vs classes 1 &amp; 3 etc) approach to calculating the metrics. Even if your accuracy is under 50%, the model may still be predicting at least one class well, so I recommend doing more analysis before making a decision.</p>
  <p>I also recommend comparing your model (if it is deep learning-based) to a traditional type of model, as deep learning usually works best with big data and with such a small dataset as you have, there may be no benefit to using deep learning (unless you are leveraging transfer learning).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T05:21:06.617Z
Answers.LastActivityDate: 2021-04-14T05:33:25.590Z
Answers.LastEditDate: 2021-04-14T05:33:25.590Z
Answers.UserDisplayName: Mike NZ
Answers.Body: <p>To know if your model needs more training data, try to plot out "learning curves", that are based on increasing size of the training set.</p>
  
  <p>Basically, you calculate training and validation accuracy metrics for 1, 2, 3, 4, 5, ..., m  training samples. Size of validation set may be constant over time. If the accuracy is still rising when your data set is fully used, then you need more training data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-29T16:52:17.167Z
Answers.LastActivityDate: 2019-07-29T17:01:08.187Z
Answers.LastEditDate: 2019-07-29T17:01:08.187Z
Answers.UserDisplayName: GKozinski
Answers.Body: <p>If data collection is expensive, it is better to first try to improve your model.</p>
  <p>You say your accuracy is bad, but have you using tried better performance metrics? A confusion matrix could help. Another potential problem is that your data may be imbalanced. What if your model is performing badly because, for example, there aren't enough samples from class 2. Now you know that to improve your model you need mode class 2 samples. This can be acquired by collecting more data, or by other class imbalance methods e.g. SMOTE.</p>
  <p>You can also check whether the performance is bad only on the test set. Is the model overfitting? Or does it perform badly on the training set too? Underfitting. 4000 samples should be enough for a neural network, but not a very large deep learning model, maybe a smaller one. Testing other machine learning models on your data is definitely a good idea. Not only you can provide your model with a benchmark, but you also may find a better model.</p>
  <p>If you think your data is good enough and your model is adequate, you can try hyperparameter optimization. This can be quire useful for getting the most out of neural networks, but it is trial and error, and it can take some time to find the best hyperparameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-03T20:07:38.797Z
Answers.LastActivityDate: 2021-05-03T20:07:38.797Z
Answers.UserDisplayName: Marcus
Body: <p>Suppose we have a data set with <span class="math-container">$4,000$</span> labeled examples. The outcome variable is trinary (three possible categorical values). Suppose the accuracy of a given model is &quot;bad&quot; (e.g. less than <span class="math-container">$50 \%$</span>).</p>
  <blockquote>
  <p><strong>Question.</strong> Should you try different traditional machine learning models (e.g. multinomial logistic regression, random forests, XGBoost,
  etc.), get more data, or try various deep learning models like
  convolutional neural networks or recurrent neural networks?</p>
  </blockquote>
  <p>If the purpose is to minimize time and effort in collecting training data, would deep learning models be a viable option over traditional machine learning models in this case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-05T10:47:15.837Z
Comments.Text: This post may need more details to be answered properly. What model were you using at the time? What were its hyper-parameters? What loss was you using?
Comments.UserDisplayName: nbro
Tags: <machine-learning><deep-learning><training><supervised-learning><labeled-datasets>
Title: If the accuracy of my current model is low ($50 \%$) and we want to minimize time in collecting more data, should we try other models?
UserDisplayName: naturalguy_12
ThreadText: # If the accuracy of my current model is low ($50 \%$) and we want to minimize time in collecting more data, should we try other models?
  
  (Asked by: username_0 on 2019-07-29)
  
  Suppose we have a data set with $4,000$ labeled examples. The outcome variable is trinary (three possible categorical values). Suppose the accuracy of a given model is "bad" (e.g. less than $50 \%$).
  
  > **Question.** Should you try different traditional machine learning models (e.g. multinomial logistic regression, random forests, XGBoost,
  > etc.), get more data, or try various deep learning models like
  > convolutional neural networks or recurrent neural networks?
  
  If the purpose is to minimize time and effort in collecting training data, would deep learning models be a viable option over traditional machine learning models in this case?
  
  **Tags:** machine-learning, deep-learning, training, supervised-learning, labeled-datasets
  > **Comments:**
  > **username_1**: This post may need more details to be answered properly. What model were you using at the time? What were its hyper-parameters? What loss was you using?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2021-04-14. Score: 2
  
  Accuracy can sometimes be a very coarse metric. When it is applied to three class problems, people often take the class label with maximum predicted probability and predict that. The probabilities of the individual labels are ignored. I'd recommend that as well as accuracy you calculate sensitivity and specificity for each class and the area under the ROC curve. For both of these, you can take a 1 vs rest (i.e. class 1 vs classes 2 & 3, class 2 vs classes 1 & 3 etc) approach to calculating the metrics. Even if your accuracy is under 50%, the model may still be predicting at least one class well, so I recommend doing more analysis before making a decision.
  
  I also recommend comparing your model (if it is deep learning-based) to a traditional type of model, as deep learning usually works best with big data and with such a small dataset as you have, there may be no benefit to using deep learning (unless you are leveraging transfer learning).
  
  
  ---
  
  ### Answer by username_3 on 2019-07-29. Score: 2
  
  To know if your model needs more training data, try to plot out "learning curves", that are based on increasing size of the training set.
  
  Basically, you calculate training and validation accuracy metrics for 1, 2, 3, 4, 5, ..., m training samples. Size of validation set may be constant over time. If the accuracy is still rising when your data set is fully used, then you need more training data.
  
  
  ---
  
  ### Answer by username_4 on 2021-05-03. Score: 0
  
  If data collection is expensive, it is better to first try to improve your model.
  
  You say your accuracy is bad, but have you using tried better performance metrics? A confusion matrix could help. Another potential problem is that your data may be imbalanced. What if your model is performing badly because, for example, there aren't enough samples from class 2. Now you know that to improve your model you need mode class 2 samples. This can be acquired by collecting more data, or by other class imbalance methods e.g. SMOTE.
  
  You can also check whether the performance is bad only on the test set. Is the model overfitting? Or does it perform badly on the training set too? Underfitting. 4000 samples should be enough for a neural network, but not a very large deep learning model, maybe a smaller one. Testing other machine learning models on your data is definitely a good idea. Not only you can provide your model with a benchmark, but you also may find a better model.
  
  If you think your data is good enough and your model is adequate, you can try hyperparameter optimization. This can be quire useful for getting the most out of neural networks, but it is trial and error, and it can take some time to find the best hyperparameters.
  
  
  ---
  
  