Answers.Body: <p>In general, the larger the training set, the better. See <a href="http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf" rel="nofollow">The Unreasonable effectiveness of Data</a>, though this article is quite dated (written in 2009). Xavier Amatriain, a researcher at Netflix has a <a href="https://www.quora.com/In-machine-learning-is-more-data-always-better-than-better-algorithms" rel="nofollow">Quora answer</a> where he discusses that more data can sometimes hurt algorithms. </p>
  
  <p>For deep neural networks in particular, it does not seem that we have hit these limits yet. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T16:15:38.400Z
Answers.LastActivityDate: 2016-08-04T16:15:38.400Z
Answers.UserDisplayName: Harsh
Answers.Body: <p>For a finite value to be 'optimal,' typically you need some benefit from more paired up with some cost for more, and eventually the lines cross because the benefit decreases and the cost increases.</p>
  
  <p>Most models will have a reduction in error with more training data, that asymptotically approaches the best the model can do. See this image (from <a href="http://blog.revolutionanalytics.com/2015/09/why-big-data-learning-curves.html" rel="nofollow noreferrer">here</a>) as an example:</p>
  
  <p><a href="https://i.sstatic.net/Jx1AZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Jx1AZ.png" alt="Decreasing error with increasing training set size"></a></p>
  
  <p>The costs of training data are also somewhat obvious; data is costly to obtain, to store, and to move. (Assuming model complexity stays constant, the actual cost of storing, moving, and using the model remains the same, since the weights in the model are just being tuned.)</p>
  
  <p>So at some point the slope of the error-reduction curve becomes horizontal enough that more data points are costlier than they're worth, and that's the optimal amount of training data.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-04T16:09:38.327Z
Answers.LastActivityDate: 2016-08-04T16:09:38.327Z
Answers.UserDisplayName: Matthew Gray
Body: <p><a href="https://cs.stackexchange.com/a/60535/54605">At a related question in Computer Science SE</a>, a user told:</p>
  <blockquote>
  <p>Neural networks typically require a large training set.</p>
  </blockquote>
  <p>Is there a way to define the boundaries of the &quot;optimal&quot; size of a training set in the general case?</p>
  <p>When I was learning about fuzzy logic, I've heard some rules of thumb that involved examining the mathematical composition of the problem and using that to define the number of fuzzy sets.</p>
  <p>Is there such a method that can be applicable for an already defined neural network architecture?</p>
  
Tags: <neural-networks><training><optimization><sample-complexity>
Title: Is there a way to define the boundaries of the optimal size of a training set?
UserDisplayName: Zoltán Schmidt
ThreadText: # Is there a way to define the boundaries of the optimal size of a training set?
  
  (Asked by: username_0 on 2016-08-04)
  
  [At a related question in Computer Science SE](https://cs.stackexchange.com/a/60535/54605), a user told:
  
  > Neural networks typically require a large training set.
  
  Is there a way to define the boundaries of the "optimal" size of a training set in the general case?
  
  When I was learning about fuzzy logic, I've heard some rules of thumb that involved examining the mathematical composition of the problem and using that to define the number of fuzzy sets.
  
  Is there such a method that can be applicable for an already defined neural network architecture?
  
  **Tags:** neural-networks, training, optimization, sample-complexity
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2016-08-04. Score: 3 | Accepted answer
  
  For a finite value to be 'optimal,' typically you need some benefit from more paired up with some cost for more, and eventually the lines cross because the benefit decreases and the cost increases.
  
  Most models will have a reduction in error with more training data, that asymptotically approaches the best the model can do. See this image (from [here](http://blog.revolutionanalytics.com/2015/09/why-big-data-learning-curves.html)) as an example:
  
  [![Decreasing error with increasing training set size](https://i.sstatic.net/Jx1AZ.png)](https://i.sstatic.net/Jx1AZ.png)
  
  The costs of training data are also somewhat obvious; data is costly to obtain, to store, and to move. (Assuming model complexity stays constant, the actual cost of storing, moving, and using the model remains the same, since the weights in the model are just being tuned.)
  
  So at some point the slope of the error-reduction curve becomes horizontal enough that more data points are costlier than they're worth, and that's the optimal amount of training data.
  
  
  ---
  
  ### Answer by username_1 on 2016-08-04. Score: 1
  
  In general, the larger the training set, the better. See [The Unreasonable effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf), though this article is quite dated (written in 2009). Xavier Amatriain, a researcher at Netflix has a [Quora answer](https://www.quora.com/In-machine-learning-is-more-data-always-better-than-better-algorithms) where he discusses that more data can sometimes hurt algorithms.
  
  For deep neural networks in particular, it does not seem that we have hit these limits yet.
  
  
  ---
  
  