Answers.Body: <p>Generally speaking, the number of training examples needed to train a neural network for your classification task cannot be determined <em>a priori</em>.  Generalizations such as the claim that 10K training examples needed are obtuse.  This question is complicated, but here are some considerations:</p>
  <ol>
  <li>Technically, you can generate a model with very few training
  examples (equal to the number of classes), but model performance
  will be poor.  Thus, it is not sufficient to ask how
  many training examples are needed.  It would be more apropos to ask
  how many are needed to achieve a target performance level.</li>
  <li>The number of examples depends on the complexity of your task, the expressivity of your model, and the quality of your data.</li>
  </ol>
  <ul>
  <li>How many classes do you have (binary OR 100 classes)?</li>
  <li>How different are the classes from each other (snake vs plane OR python vs boa)?</li>
  <li>What types of features do you have (imaging, time series, tabular data)?</li>
  <li>How much label noise is in the data?</li>
  <li>How much feature noise is in the data?</li>
  <li>How predictive are the features of the classes (relates to #1)?</li>
  <li>What is the capacity (model complexity) of your model to learn the pattern?</li>
  </ul>
  <ol start="3">
  <li>In a more practical sense, I search the literature for publications that worked on similar tasks as my task and see what models they used, how many examples they used, and what type of performance they achieved.</li>
  <li>Consider expanding your training data with Data Augmentation or GANs or public data sets or working with collaborators.</li>
  <li>Consider transfer learning if you have scarce data.</li>
  <li>Besides neural networks, there are again no <em>a priori</em> methods for determining which traditional ML model is going to perform better.  You have to empirically test them out.</li>
  <li>Consider N-shot learning.</li>
  <li>Depending on whether you have very little data over all or whether you have very little labeled data, considered semi-supervised learning.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-04T21:15:43.287Z
Answers.Text: I agree with most! Only a minor point: data augmentation can be harmful. Only in the setting of multi objective optimization (like optimizing e.g. recall of multiple classes etc.) data augmentation is a very indirect way of changing the tradeoffs
Answers.UserDisplayName: Ggjj11
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-11T14:22:15.700Z
Answers.LastActivityDate: 2022-12-11T16:09:20.600Z
Answers.LastEditDate: 2022-12-11T16:09:20.600Z
Answers.UserDisplayName: Snehal Patel
Body: <p>It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification problem. However, you don't always have lots of data to train your machine learning algorithm.</p>
  <p>Which classical machine learning methods work well with little data? I'm thinking of things like KNN, linear regression, support vector machines, random forests, etc.</p>
  <p>Is there a paper that systematically investigates machine learning methods when data are scarse? If not, are there some rules of thumb one can follow?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-16T21:32:07.813Z
Comments.Text: Currently to train state-of-the-art robots for bicycling needs millions of trials from a blank slate on average compared with much less trials for average young aged human. You may contemplate the root cause here. There seems no other way around this issue until we have true AGI to have agent continuously evolved to learn new objectives/rewards based upon previous learned policies in the RL formulation.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-04T21:12:50.547Z
Comments.Text: You are asking for sample efficiency of learning algorithms
Comments.UserDisplayName: Ggjj11
Tags: <machine-learning><deep-learning><reference-request><datasets><algorithm-request>
Title: Machine Learning Methods commonly used when data are scarse
UserDisplayName: postnubilaphoebus
ThreadText: # Machine Learning Methods commonly used when data are scarse
  
  (Asked by: username_0 on 2022-11-15)
  
  It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification problem. However, you don't always have lots of data to train your machine learning algorithm.
  
  Which classical machine learning methods work well with little data? I'm thinking of things like KNN, linear regression, support vector machines, random forests, etc.
  
  Is there a paper that systematically investigates machine learning methods when data are scarse? If not, are there some rules of thumb one can follow?
  
  **Tags:** machine-learning, deep-learning, reference-request, datasets, algorithm-request
  > **Comments:**
  > **username_1**: Currently to train state-of-the-art robots for bicycling needs millions of trials from a blank slate on average compared with much less trials for average young aged human. You may contemplate the root cause here. There seems no other way around this issue until we have true AGI to have agent continuously evolved to learn new objectives/rewards based upon previous learned policies in the RL formulation.  
  > **username_2**: You are asking for sample efficiency of learning algorithms  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-12-11. Score: 1
  
  Generally speaking, the number of training examples needed to train a neural network for your classification task cannot be determined *a priori*. Generalizations such as the claim that 10K training examples needed are obtuse. This question is complicated, but here are some considerations:
  
  1. Technically, you can generate a model with very few training
     examples (equal to the number of classes), but model performance
     will be poor. Thus, it is not sufficient to ask how
     many training examples are needed. It would be more apropos to ask
     how many are needed to achieve a target performance level.
  2. The number of examples depends on the complexity of your task, the expressivity of your model, and the quality of your data.
  
  * How many classes do you have (binary OR 100 classes)?
  * How different are the classes from each other (snake vs plane OR python vs boa)?
  * What types of features do you have (imaging, time series, tabular data)?
  * How much label noise is in the data?
  * How much feature noise is in the data?
  * How predictive are the features of the classes (relates to #1)?
  * What is the capacity (model complexity) of your model to learn the pattern?
  
  3. In a more practical sense, I search the literature for publications that worked on similar tasks as my task and see what models they used, how many examples they used, and what type of performance they achieved.
  4. Consider expanding your training data with Data Augmentation or GANs or public data sets or working with collaborators.
  5. Consider transfer learning if you have scarce data.
  6. Besides neural networks, there are again no *a priori* methods for determining which traditional ML model is going to perform better. You have to empirically test them out.
  7. Consider N-shot learning.
  8. Depending on whether you have very little data over all or whether you have very little labeled data, considered semi-supervised learning.
  
  > **Comments:**
  > **username_2**: I agree with most! Only a minor point: data augmentation can be harmful. Only in the setting of multi objective optimization (like optimizing e.g. recall of multiple classes etc.) data augmentation is a very indirect way of changing the tradeoffs  
  
  ---
  
  