Answers.Body: <p>Consider the LCD (least common denominator) principle in algebra. A larger denominator would work for most processes for which the LCD would be calculated, however the least is the one used by convention.  Why?  The interest in prime numbers in general is based on the value of reductive methodologies.</p>
  
  <p>In philosophy, Occam's Razor is a principle that, given two conceptual constructs that correlate equally well with observations, the simpler is most likely the best.  A more formal and generalized prescription is this:</p>
  
  <blockquote>
    <p>Given two mathematical models of a physical system with equal correlation to the set of all observations about the system being modeled, the simpler model is more likely to be the most likely to be predictive of conditions outside those of the observation set.</p>
  </blockquote>
  
  <p>This principle of simplicity as the functional ideal is true of decision trees.  The more likely functional and clear decision tree resulting from a given data set driving its construction will be the simplest.  Without a clear reason for additional complexity, there may be no benefit derived from the complexity added and yet there may be penalties in terms of clarity and therefore maintainability and verifiability. There may be computational penalties too, in some applications.</p>
  
  <p>The question mentions, "Bigger extensions of the tree which could in principle perform better," however the performance of the tree should be a matter optimization in preparing for execution and execution of the decision tree in real time.  In other words, the minimalist decision tree is the most clear, workable, and verifiable construct, however a clever software engineer could translate that minimalist construct to a run time optimized equivalent.</p>
  
  <p>Just as with compilation of source code, performance is multidimensional in meaning.  There are time efficiency, memory efficiency, network bandwidth efficiency, or other performance metrics that could be used when optimizing the tree for run time.  Nonetheless, the simpler tree is the best starting point for any weighted combination of these interests.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-03T20:37:27.267Z
Answers.LastActivityDate: 2017-08-03T20:37:27.267Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>The bigger your tree is the more overfitting your model is. In machine learning, we always prefer a simpler model unless there is good reason to go for complication.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-31T13:30:27.757Z
Answers.Text: Why would a bigger tree correspond to more overfitting? i don't see any logic behind it.
Answers.UserDisplayName: imflash217
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-31T18:42:53.983Z
Answers.Text: More complexity = more potential for noise and errors.  Although I'm not working in Machine Learning specifically, this is consistent with the [KISS](https://en.wikipedia.org/wiki/KISS_principle) principle ingrained in me by the punch-card generation.  From a statistical standpoint, it sounds like, in this case, more complexity runs a large risk of reducing the reliability and value of the results.
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-01T02:46:31.780Z
Answers.Text: @DukeZhou Correct. A big tree is more likely to be complicated and less likely be useful for prediction.
Answers.UserDisplayName: SmallChess
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-01T18:53:18.567Z
Answers.Text: To me this does not make any valid reason. Why wouldn't i choose a bigger Decision Tree if it performs far far better than a shorter tree! There should be some mathematical foundation for it. Also, even if what you are saying is correct then there must be some foundation/validation of that. I couldn't find any.
Answers.UserDisplayName: imflash217
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-01T22:54:32.547Z
Answers.Text: @vinaykumar2491 please read a machine learning book carefully. What you ask is just beginner concepts. Pointless for me to explain. You need to put some efforts.
Answers.UserDisplayName: SmallChess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-27T18:39:50.517Z
Answers.Text: For some, the concept of overfitting by complexity may be simple at first glance, but the reason why can go as deep as Occam's Razor principle in philosophy.
Answers.UserDisplayName: Toon Tran
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-31T13:25:49.520Z
Answers.LastActivityDate: 2017-07-31T13:25:49.520Z
Answers.UserDisplayName: SmallChess
Answers.Body: <p>Adding to SmallChess's answer , 
  Larger trees(with many nodes) are too adapted to the training set, as a small change in the input train data might cause the trees to change very much and hence change the estimate value too much.This is mainly due to the hierarchical structures of trees(because a change in a higher node may cause all lower nodes to change).
  As an extreme case you can think of a large tree where in each training example has its own node.Such a is absolutely useless for test prediction.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-01T19:10:38.050Z
Answers.Text: But, when we change the **training dataset** there is always a possibility of change in the tree structure (as we would not be knowing what the final outcome of new training will be). And if the new Tree performs better in terms of accuracy, time/space complexity etc. then why should not we choose it.
Answers.UserDisplayName: imflash217
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-02T04:01:00.943Z
Answers.Text: @vinaykumar2491 It's all about variance-bias tradeoff. I recommend a beginner machine learning book, otherwise you don't really know what you're talking about. You'll understand your question is not really a question.
Answers.UserDisplayName: SmallChess
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-31T14:13:56.340Z
Answers.LastActivityDate: 2017-07-31T14:13:56.340Z
Answers.UserDisplayName: Fenil
Body: <p>I have been following the <a href="http://www.cs.cmu.edu/~ninamf/courses/601sp15/lectures.shtml" rel="noreferrer">ML course by Tom Mitchel</a>. 
  The inherent assumption while using Decision Tree Learning Algo is: <strong>The algo. preferably chooses a Decision Tree which is the smallest.</strong> </p>
  
  <p><strong>Why is this so</strong> when we can have bigger extensions of the tree which could in principle perform better than the shorter tree?</p>
  
Tags: <machine-learning><unsupervised-learning><learning-algorithms>
Title: Why do Decision Tree Learning Algorithm preferably outputs the smallest Decision Tree?
UserDisplayName: imflash217
ThreadText: # Why do Decision Tree Learning Algorithm preferably outputs the smallest Decision Tree?
  
  (Asked by: username_0 on 2017-07-31)
  
  I have been following the [ML course by Tom Mitchel](http://www.cs.cmu.edu/~ninamf/courses/601sp15/lectures.shtml).
  The inherent assumption while using Decision Tree Learning Algo is: **The algo. preferably chooses a Decision Tree which is the smallest.**
  
  **Why is this so** when we can have bigger extensions of the tree which could in principle perform better than the shorter tree?
  
  **Tags:** machine-learning, unsupervised-learning, learning-algorithms
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2017-08-03. Score: 0 | Accepted answer
  
  Consider the LCD (least common denominator) principle in algebra. A larger denominator would work for most processes for which the LCD would be calculated, however the least is the one used by convention. Why? The interest in prime numbers in general is based on the value of reductive methodologies.
  
  In philosophy, Occam's Razor is a principle that, given two conceptual constructs that correlate equally well with observations, the simpler is most likely the best. A more formal and generalized prescription is this:
  
  > Given two mathematical models of a physical system with equal correlation to the set of all observations about the system being modeled, the simpler model is more likely to be the most likely to be predictive of conditions outside those of the observation set.
  
  This principle of simplicity as the functional ideal is true of decision trees. The more likely functional and clear decision tree resulting from a given data set driving its construction will be the simplest. Without a clear reason for additional complexity, there may be no benefit derived from the complexity added and yet there may be penalties in terms of clarity and therefore maintainability and verifiability. There may be computational penalties too, in some applications.
  
  The question mentions, "Bigger extensions of the tree which could in principle perform better," however the performance of the tree should be a matter optimization in preparing for execution and execution of the decision tree in real time. In other words, the minimalist decision tree is the most clear, workable, and verifiable construct, however a clever software engineer could translate that minimalist construct to a run time optimized equivalent.
  
  Just as with compilation of source code, performance is multidimensional in meaning. There are time efficiency, memory efficiency, network bandwidth efficiency, or other performance metrics that could be used when optimizing the tree for run time. Nonetheless, the simpler tree is the best starting point for any weighted combination of these interests.
  
  
  ---
  
  ### Answer by username_2 on 2017-07-31. Score: 2
  
  The bigger your tree is the more overfitting your model is. In machine learning, we always prefer a simpler model unless there is good reason to go for complication.
  
  > **Comments:**
  > **username_0**: Why would a bigger tree correspond to more overfitting? i don't see any logic behind it.  
  > **username_3**: More complexity = more potential for noise and errors.  Although I'm not working in Machine Learning specifically, this is consistent with the [KISS](https://en.wikipedia.org/wiki/KISS_principle) principle ingrained in me by the punch-card generation.  From a statistical standpoint, it sounds like, in this case, more complexity runs a large risk of reducing the reliability and value of the results.  
  > **username_2**: @username_3 Correct. A big tree is more likely to be complicated and less likely be useful for prediction.  
  > **username_0**: To me this does not make any valid reason. Why wouldn't i choose a bigger Decision Tree if it performs far far better than a shorter tree! There should be some mathematical foundation for it. Also, even if what you are saying is correct then there must be some foundation/validation of that. I couldn't find any.  
  > **username_2**: @vinaykumar2491 please read a machine learning book carefully. What you ask is just beginner concepts. Pointless for me to explain. You need to put some efforts.  
  > **username_4**: For some, the concept of overfitting by complexity may be simple at first glance, but the reason why can go as deep as Occam's Razor principle in philosophy.  
  
  ---
  
  ### Answer by username_5 on 2017-07-31. Score: 2
  
  Adding to username_2's answer ,
  Larger trees(with many nodes) are too adapted to the training set, as a small change in the input train data might cause the trees to change very much and hence change the estimate value too much.This is mainly due to the hierarchical structures of trees(because a change in a higher node may cause all lower nodes to change).
  As an extreme case you can think of a large tree where in each training example has its own node.Such a is absolutely useless for test prediction.
  
  > **Comments:**
  > **username_0**: But, when we change the **training dataset** there is always a possibility of change in the tree structure (as we would not be knowing what the final outcome of new training will be). And if the new Tree performs better in terms of accuracy, time/space complexity etc. then why should not we choose it.  
  > **username_2**: @vinaykumar2491 It's all about variance-bias tradeoff. I recommend a beginner machine learning book, otherwise you don't really know what you're talking about. You'll understand your question is not really a question.  
  
  ---
  
  