Answers.Body: <p>I will try to answer all three questions to the best of my ability.</p>
  
  <p><strong>Is this basically true? Quick googling just brought me to a lot of papers trying to fit decision trees into incremental learning.</strong></p>
  
  <p>The problem with decision trees in an online learning setting is that the model should be able to update when experiences (i.e. state, action, reward, new state) of the environment are collected. Any data that does not reflect the structure of the tree will cause the decision tree to fall apart and requires you to rebuild the model again. This is not necessarily a problem, but it is something you have to take into consideration when building a model.</p>
  
  <p><strong>What other algorithms fall under this category?</strong></p>
  
  <p>Other algorithms which assumes that the data is known beforehand (offline). For example K-NN or Learning Vector Quantization (LVQ). However it should be noted that these algorithms can be adapted to work in an online fashion.</p>
  
  <p><strong>Are neural nets good for incremental learning? What other algorithms would be good?</strong></p>
  
  <p>Neural networks can capture changes from online experiences fairly well. The reason is that each weight in such a network is adapted when a new experience comes in. The weights are adapted with regards to an error function and this allows to update the model relatively easy. </p>
  
  <p>In 2016, Venkatesan, et al proposed a novel progressive learning technique for multi-class classification. You might want to look into this as the system that they use allows for online learning. Other techniques exists as well and should be found when googling for online learning methods. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-19T09:52:41.057Z
Answers.Text: Some decision trees are built for online learning! Check "incremental decision trees" on Wikipedia or the paper "Mining High-speed data streams" from Domingos & Hulten, which presents Very Fast Decision Trees (VFDTs). They are built to keep learning as the data arrives, which is exactly the online learning setting.
Answers.UserDisplayName: Adrien Luxey
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-28T23:00:54.607Z
Answers.LastActivityDate: 2017-10-28T23:00:54.607Z
Answers.UserDisplayName: Daemonstool
Body: <p>I can recall that a professor once said that decision trees are not good for incremental learning, as they have to be rebuilt from the ground up if new training examples arrive.</p>
  
  <ul>
  <li>Is this basically true? Quick googling just brought me to a lot of papers trying to fit decision trees into incremental learning</li>
  <li>What other algorithms fall under this category?</li>
  <li>Are Neural Nets good for incremental learning? What other algorithms would be good?</li>
  </ul>
  
Tags: <machine-learning><reinforcement-learning><learning-algorithms>
Title: Is a decision tree less suitable for incremental learning than e.g. a neural net?
UserDisplayName: Ulu83
ThreadText: # Is a decision tree less suitable for incremental learning than e.g. a neural net?
  
  (Asked by: username_0 on 2017-08-16)
  
  I can recall that a professor once said that decision trees are not good for incremental learning, as they have to be rebuilt from the ground up if new training examples arrive.
  
  * Is this basically true? Quick googling just brought me to a lot of papers trying to fit decision trees into incremental learning
  * What other algorithms fall under this category?
  * Are Neural Nets good for incremental learning? What other algorithms would be good?
  
  **Tags:** machine-learning, reinforcement-learning, learning-algorithms
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-28. Score: 4 | Accepted answer
  
  I will try to answer all three questions to the best of my ability.
  
  **Is this basically true? Quick googling just brought me to a lot of papers trying to fit decision trees into incremental learning.**
  
  The problem with decision trees in an online learning setting is that the model should be able to update when experiences (i.e. state, action, reward, new state) of the environment are collected. Any data that does not reflect the structure of the tree will cause the decision tree to fall apart and requires you to rebuild the model again. This is not necessarily a problem, but it is something you have to take into consideration when building a model.
  
  **What other algorithms fall under this category?**
  
  Other algorithms which assumes that the data is known beforehand (offline). For example K-NN or Learning Vector Quantization (LVQ). However it should be noted that these algorithms can be adapted to work in an online fashion.
  
  **Are neural nets good for incremental learning? What other algorithms would be good?**
  
  Neural networks can capture changes from online experiences fairly well. The reason is that each weight in such a network is adapted when a new experience comes in. The weights are adapted with regards to an error function and this allows to update the model relatively easy.
  
  In 2016, Venkatesan, et al proposed a novel progressive learning technique for multi-class classification. You might want to look into this as the system that they use allows for online learning. Other techniques exists as well and should be found when googling for online learning methods.
  
  > **Comments:**
  > **username_2**: Some decision trees are built for online learning! Check "incremental decision trees" on Wikipedia or the paper "Mining High-speed data streams" from Domingos & Hulten, which presents Very Fast Decision Trees (VFDTs). They are built to keep learning as the data arrives, which is exactly the online learning setting.  
  
  ---
  
  