Answers.Body: <p>What you want to calculate/estimate is known as the <a href="https://en.wikipedia.org/wiki/Sample_complexity" rel="noreferrer"><strong>sample complexity</strong></a> in <a href="https://ai.stackexchange.com/a/20358/2444">computational learning theory</a>. If you knew the VC dimension of the neural network, you may be able to estimate the sample complexity, but <a href="https://ai.stackexchange.com/a/20398/2444">your estimate (bound) may not be very tight anyway</a> (maybe because the estimate of the VC dimension is also not tight). <a href="https://ai.stackexchange.com/a/17881/2444">Here</a> is more info about the VC dimension and the VC dimension of a few neural networks.</p>
  <p>If we were able to easily compute (an accurate estimate of) the sample complexity in all cases, this would be extremely useful, as we could exclude (or not) that the size of the training dataset is the problem of <a href="https://www.ibm.com/cloud/learn/underfitting" rel="noreferrer">under-fitting/over-fitting</a>.</p>
  <p>Nowadays, I think that most people do not usually estimate the sample complexity (I could be wrong, i.e. there may be simple ways to do this approximately, apart from saying that you probably need more training data points than the number of parameters, which is what many practitioners will tell you, maybe misleadingly), but, instead, collect as much data as possible, then train the neural network with that data; if it performs <em>poorly</em> on the test dataset, then you may try to collect more data (given that it's an indication that the neural network did not learn the target function); if you see that the test performance initially increases but then decreases while the training performance is always increasing, it may be an indication of <em>over-fitting</em>, so you may want to use a <strong>regularisation</strong> technique (like <em>dropout</em>), which you should probably use anyway, especially when you have a small dataset.</p>
  <p>I also want to point out that, with the discovery of the phenomena <a href="https://openai.com/blog/deep-double-descent/" rel="noreferrer"><strong>double descent</strong></a> and <a href="https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper.pdf" rel="noreferrer">grokking</a>, these traditional guidelines or practices (e.g. <a href="https://en.wikipedia.org/wiki/Early_stopping" rel="noreferrer">early stopping</a>) may sometimes be misleading (I don't know when or why), i.e. just keep in mind that <em>sometimes</em> they may not be applicable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T08:55:21.303Z
Answers.Text: Worth adding a standard way to assess whether more data would help = plotting a learning curve where the x axis is amount of data used to train, and y axis is performance of the model on a (fixed) validation set. If it plateaus, then more data may not help (with the specific model being tested), if it is still trending upwards, more data probably will help.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-16T22:08:41.857Z
Answers.LastActivityDate: 2021-11-16T22:30:22.720Z
Answers.LastEditDate: 2021-11-16T22:30:22.720Z
Answers.UserDisplayName: nbro
Body: <p>I am working on a project where I have to classify around 1000 unique objects. I'm trying to plan how much training data I will need to collect. I was planning on using ResNet-50. Is there anyway I can estimate the amount of photos I should plan to collect ahead of time (assuming I will collect an equal amount of photos of each class)?</p>
  
Tags: <computer-vision><computational-learning-theory><training-datasets><sample-complexity>
Title: How can I estimate how many photos I need to train ResNet-50 for image classification?
UserDisplayName: Tyler Hilbert
ThreadText: # How can I estimate how many photos I need to train ResNet-50 for image classification?
  
  (Asked by: username_0 on 2021-11-16)
  
  I am working on a project where I have to classify around 1000 unique objects. I'm trying to plan how much training data I will need to collect. I was planning on using ResNet-50. Is there anyway I can estimate the amount of photos I should plan to collect ahead of time (assuming I will collect an equal amount of photos of each class)?
  
  **Tags:** computer-vision, computational-learning-theory, training-datasets, sample-complexity
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-16. Score: 6 | Accepted answer
  
  What you want to calculate/estimate is known as the [**sample complexity**](https://en.wikipedia.org/wiki/Sample_complexity) in [computational learning theory](https://ai.stackexchange.com/a/20358/2444). If you knew the VC dimension of the neural network, you may be able to estimate the sample complexity, but [your estimate (bound) may not be very tight anyway](https://ai.stackexchange.com/a/20398/2444) (maybe because the estimate of the VC dimension is also not tight). [Here](https://ai.stackexchange.com/a/17881/2444) is more info about the VC dimension and the VC dimension of a few neural networks.
  
  If we were able to easily compute (an accurate estimate of) the sample complexity in all cases, this would be extremely useful, as we could exclude (or not) that the size of the training dataset is the problem of [under-fitting/over-fitting](https://www.ibm.com/cloud/learn/underfitting).
  
  Nowadays, I think that most people do not usually estimate the sample complexity (I could be wrong, i.e. there may be simple ways to do this approximately, apart from saying that you probably need more training data points than the number of parameters, which is what many practitioners will tell you, maybe misleadingly), but, instead, collect as much data as possible, then train the neural network with that data; if it performs *poorly* on the test dataset, then you may try to collect more data (given that it's an indication that the neural network did not learn the target function); if you see that the test performance initially increases but then decreases while the training performance is always increasing, it may be an indication of *over-fitting*, so you may want to use a **regularisation** technique (like *dropout*), which you should probably use anyway, especially when you have a small dataset.
  
  I also want to point out that, with the discovery of the phenomena [**double descent**](https://openai.com/blog/deep-double-descent/) and [grokking](https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper.pdf), these traditional guidelines or practices (e.g. [early stopping](https://en.wikipedia.org/wiki/Early_stopping)) may sometimes be misleading (I don't know when or why), i.e. just keep in mind that *sometimes* they may not be applicable.
  
  > **Comments:**
  > **username_2**: Worth adding a standard way to assess whether more data would help = plotting a learning curve where the x axis is amount of data used to train, and y axis is performance of the model on a (fixed) validation set. If it plateaus, then more data may not help (with the specific model being tested), if it is still trending upwards, more data probably will help.  
  
  ---
  
  