Answers.Body: <p>Your description fits the ML terminology called <a href="https://en.wikipedia.org/wiki/Curriculum_learning" rel="nofollow noreferrer">curriculum learning</a>.</p>
  <blockquote>
  <p>curriculum learning is the technique of successively increasing the difficulty of examples in the training set that is presented to a model over multiple training iterations. This can produce better results than exposing the model to the full training set immediately under some circumstances; most typically, when the model is able to learn general principles from easier examples, and then gradually incorporate more complex and nuanced information as harder examples are introduced, such as edge cases. This has been shown to work in many domains, most likely as a form of regularization.</p>
  </blockquote>
  <p>Therefore the usual reason why you need to do training this way barring online continue or incremental learning cases or not enough memory resource cases, is that to converge to a better local optimum more efficiently if the global optimum is hard or unable to be found which is especially a common case for complex deep learning models with many training data consisting of different 'difficulty' or other characteristics, such as training on High School Math Olympiad Q&amp;A dataset <a href="https://github.com/openai/miniF2F" rel="nofollow noreferrer">miniF2F</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-13T00:18:21.600Z
Answers.Text: I don't think that curriculum learning and incremental learning are synonymous. Incremental learning is more about continual or lifelong learning, i.e. learning continuously (i.e. not just once) without forgetting - the task might not change but it could. What the user describes could refer to both curriculum learning or incremental learning I think. Curriculum learning is about simplifying the original task and increasing its difficulty with the goal to make the model learn first a simple task, then a more complicated one.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-13T00:25:36.160Z
Answers.Text: @nbro thx for your reading and critique. I've already envisioned and used 'sometimes' to hedge my invocation of the non-standard term 'incremental learning', which is a fluid and flexible terminology anyway, especially in ML vast and oftentimes confusing term base. I've also chosen the term 'continue learning' in the beginning of my last paragraph to describe your similar idea. But here OP seems not about general continue learning, and I've also explained the main reason why we need such staged training barring the said two obvious exceptions. Hope this clarifies to the questioner as well.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-13T00:33:04.690Z
Answers.Text: If you can't find a reputable source that states that both are synonymous, I'd just remove the statement that claims that curriculum learning is also known as incremental learning. I've never heard the term "staged training". This must be a relatively recent term. Anyway, for the curriculum learning, see e.g section 4 of https://arxiv.org/pdf/1410.4615. We could loosely say that curriculum learning is "incremental", but I think both have distinct meanings usually. But you're right that terms might be overloaded.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-13T00:41:56.207Z
Answers.Text: @nbro sure, I've removed incremental learning per your suggestion to reduce unnecessary confusion.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-12T19:56:21.963Z
Answers.LastActivityDate: 2024-12-13T00:40:05.897Z
Answers.LastEditDate: 2024-12-13T00:40:05.897Z
Answers.UserDisplayName: cinch
Body: <p>When training a neural network in supervised learning, the common approach is to train on the whole training set.</p>
  <p>However, it is technically possible to use only 50% of the training set, start training, then after a given number of epochs, add 10% of the training set for instance, continue training, add 10% more and so on.</p>
  <p>Is this a well-known technique and if so, does it have a name?</p>
  
Tags: <machine-learning><training><terminology>
Title: How do we call the technique of increasing the training set little by little during training?
UserDisplayName: anderstood
ThreadText: # How do we call the technique of increasing the training set little by little during training?
  
  (Asked by: username_0 on 2024-12-12)
  
  When training a neural network in supervised learning, the common approach is to train on the whole training set.
  
  However, it is technically possible to use only 50% of the training set, start training, then after a given number of epochs, add 10% of the training set for instance, continue training, add 10% more and so on.
  
  Is this a well-known technique and if so, does it have a name?
  
  **Tags:** machine-learning, training, terminology
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-12. Score: 1
  
  Your description fits the ML terminology called [curriculum learning](https://en.wikipedia.org/wiki/Curriculum_learning).
  
  > curriculum learning is the technique of successively increasing the difficulty of examples in the training set that is presented to a model over multiple training iterations. This can produce better results than exposing the model to the full training set immediately under some circumstances; most typically, when the model is able to learn general principles from easier examples, and then gradually incorporate more complex and nuanced information as harder examples are introduced, such as edge cases. This has been shown to work in many domains, most likely as a form of regularization.
  
  Therefore the usual reason why you need to do training this way barring online continue or incremental learning cases or not enough memory resource cases, is that to converge to a better local optimum more efficiently if the global optimum is hard or unable to be found which is especially a common case for complex deep learning models with many training data consisting of different 'difficulty' or other characteristics, such as training on High School Math Olympiad Q&A dataset [miniF2F](https://github.com/openai/miniF2F).
  
  > **Comments:**
  > **username_2**: I don't think that curriculum learning and incremental learning are synonymous. Incremental learning is more about continual or lifelong learning, i.e. learning continuously (i.e. not just once) without forgetting - the task might not change but it could. What the user describes could refer to both curriculum learning or incremental learning I think. Curriculum learning is about simplifying the original task and increasing its difficulty with the goal to make the model learn first a simple task, then a more complicated one.  
  > **username_1**: @nbro thx for your reading and critique. I've already envisioned and used 'sometimes' to hedge my invocation of the non-standard term 'incremental learning', which is a fluid and flexible terminology anyway, especially in ML vast and oftentimes confusing term base. I've also chosen the term 'continue learning' in the beginning of my last paragraph to describe your similar idea. But here OP seems not about general continue learning, and I've also explained the main reason why we need such staged training barring the said two obvious exceptions. Hope this clarifies to the questioner as well.  
  > **username_2**: If you can't find a reputable source that states that both are synonymous, I'd just remove the statement that claims that curriculum learning is also known as incremental learning. I've never heard the term "staged training". This must be a relatively recent term. Anyway, for the curriculum learning, see e.g section 4 of https://arxiv.org/pdf/1410.4615. We could loosely say that curriculum learning is "incremental", but I think both have distinct meanings usually. But you're right that terms might be overloaded.  
  > **username_1**: @nbro sure, I've removed incremental learning per your suggestion to reduce unnecessary confusion.  
  
  ---
  
  