Answers.Body: <p>The concept of <a href="https://en.wikipedia.org/wiki/Decision_tree_pruning" rel="nofollow noreferrer">pruning a tree</a> is mainly to avoid overfitting in either decision tree (DT) classifier or regressor learning when there is noise in the data or the number of training samples is too small to produce a representative sample of the true target function.</p>
  <p>Furthermore DT regressor usually needs to transform possibly many features' continuous ranges to dynamically created possibly multiple <em>ad hoc</em> Boolean test attributes based on their information gains without firm theoretical basis, and each of its leaf nodes contains a <em>constant</em> value that serves as the predicted value which is typically the mean or median of the target values of the training data in each leaf node partition, therefore this <em>effectively</em> creates a <em>noisy</em> training dataset even the untransformed raw dataset is noise free. Thus for regression problem DT learning is very prone to overfitting and the concept of pruning becomes extremely important to reduce its inherent high variance and MSE/R-squared of its performance metrics.</p>
  <p>Of course the learning algo can stop growing the tree earlier before it reaches the point where it perfectly regresses all the training data, but it's difficult to estimate precisely the minimum tree depth or leaf nodes.</p>
  <blockquote>
  <p>Prepruning methods share a common problem, the horizon effect. This is to be understood as the undesired premature termination of the induction by the stop () criterion.</p>
  </blockquote>
  <p>Thus post-pruning (or just pruning) is the most common way of simplifying trees. For DT regressors you just replace below classification accuracy confusions of validation data with MSE or R-squared</p>
  <blockquote>
  <p>Here, nodes and subtrees are replaced with leaves to reduce complexity. Pruning can not only significantly reduce the size but also improve the classification accuracy of unseen objects... The procedures are differentiated on the basis of their approach in the tree (top-down or bottom-up)... One of the simplest forms of pruning is reduced error pruning. Starting at the leaves, each node is replaced with its most popular class. If the prediction accuracy is not affected then the change is kept. While somewhat naive, reduced error pruning has the advantage of simplicity and speed.</p>
  </blockquote>
  <p>For a detailed pioneering example of the popular reduced error pruning see <em>Quinlan, J. R. (1987), 'Simplifying decision trees', International Journal of Man-Machine Studies 27, 221-234.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-05T06:30:30.763Z
Answers.LastActivityDate: 2023-11-10T00:39:05.187Z
Answers.LastEditDate: 2023-11-10T00:39:05.187Z
Answers.UserDisplayName: cinch
Body: <p>What is the concept of pruning a tree in Machine Learning regression problems? I am confused and a simple explanation would be great.</p>
  
Tags: <machine-learning><definitions><decision-trees>
Title: What is the concept of pruning a tree in Machine Learning regression problems?
UserDisplayName: Shekhar
ThreadText: # What is the concept of pruning a tree in Machine Learning regression problems?
  
  (Asked by: username_0 on 2023-11-05)
  
  What is the concept of pruning a tree in Machine Learning regression problems? I am confused and a simple explanation would be great.
  
  **Tags:** machine-learning, definitions, decision-trees
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-05. Score: 1
  
  The concept of [pruning a tree](https://en.wikipedia.org/wiki/Decision_tree_pruning) is mainly to avoid overfitting in either decision tree (DT) classifier or regressor learning when there is noise in the data or the number of training samples is too small to produce a representative sample of the true target function.
  
  Furthermore DT regressor usually needs to transform possibly many features' continuous ranges to dynamically created possibly multiple *ad hoc* Boolean test attributes based on their information gains without firm theoretical basis, and each of its leaf nodes contains a *constant* value that serves as the predicted value which is typically the mean or median of the target values of the training data in each leaf node partition, therefore this *effectively* creates a *noisy* training dataset even the untransformed raw dataset is noise free. Thus for regression problem DT learning is very prone to overfitting and the concept of pruning becomes extremely important to reduce its inherent high variance and MSE/R-squared of its performance metrics.
  
  Of course the learning algo can stop growing the tree earlier before it reaches the point where it perfectly regresses all the training data, but it's difficult to estimate precisely the minimum tree depth or leaf nodes.
  
  > Prepruning methods share a common problem, the horizon effect. This is to be understood as the undesired premature termination of the induction by the stop () criterion.
  
  Thus post-pruning (or just pruning) is the most common way of simplifying trees. For DT regressors you just replace below classification accuracy confusions of validation data with MSE or R-squared
  
  > Here, nodes and subtrees are replaced with leaves to reduce complexity. Pruning can not only significantly reduce the size but also improve the classification accuracy of unseen objects... The procedures are differentiated on the basis of their approach in the tree (top-down or bottom-up)... One of the simplest forms of pruning is reduced error pruning. Starting at the leaves, each node is replaced with its most popular class. If the prediction accuracy is not affected then the change is kept. While somewhat naive, reduced error pruning has the advantage of simplicity and speed.
  
  For a detailed pioneering example of the popular reduced error pruning see *Quinlan, J. R. (1987), 'Simplifying decision trees', International Journal of Man-Machine Studies 27, 221-234.*
  
  
  ---
  
  