Answers.Body: <p>I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal for example has done lots of work on this.</p>
  <p>In your decision tree example, I believe you're talking about pruning, which is different. In that context you're removing nodes that you <em>know</em> aren't contributing. In dropout, you randomly turn off nodes during training in order to prevent individual nodes from being too influential, but the nodes are never removed.</p>
  <p>You might also be interested in L1 regularization in parameterized models. This is when you add a penalty according to the absolute weights (rather than square weights), which tends to drive less useful weights to 0. Then you can remove the nodes with almost no weight. This is more akin to your decision tree example rather than dropout though.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T13:12:52.833Z
Answers.Text: It may be also worth mentioning "pruning" in neural networks, as the OP may come across it too.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T13:18:24.957Z
Answers.Text: Ah good point, I forgot about that. Admittedly I don't know much about neural net pruning, I would've guessed it's similar to l1.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T16:34:56.290Z
Answers.Text: xgboost (a decision-tree based algorithm) does use random feature selection and random row selection for different trees, which is more similar to dropout than pruning.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T20:09:25.223Z
Answers.Text: When Random Forest is constructed features of each individual tree are selected at Random, (or in other words dropped out at random), this is done so that the overall model is not too dependent on one feature. This is pretty similar to dropout in Neural Networks.
Answers.UserDisplayName: Akavall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-18T13:07:53.047Z
Answers.LastActivityDate: 2021-07-18T13:07:53.047Z
Answers.UserDisplayName: harwiltz
Body: <p>In one Udemy course was mentioned that &quot;dropout is unique to neural networks&quot;. However, I remember an example of decision trees where nodes that are not participating in the overall result are removed, and I think that this technique is also called &quot;dropout&quot;. Am I correct?</p>
  
Tags: <neural-networks><regularization><decision-trees><dropout>
Title: Is the dropout technique specific only to neural networks?
UserDisplayName: bridgemnc
ThreadText: # Is the dropout technique specific only to neural networks?
  
  (Asked by: username_0 on 2021-07-18)
  
  In one Udemy course was mentioned that "dropout is unique to neural networks". However, I remember an example of decision trees where nodes that are not participating in the overall result are removed, and I think that this technique is also called "dropout". Am I correct?
  
  **Tags:** neural-networks, regularization, decision-trees, dropout
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-18. Score: 4 | Accepted answer
  
  I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal for example has done lots of work on this.
  
  In your decision tree example, I believe you're talking about pruning, which is different. In that context you're removing nodes that you *know* aren't contributing. In dropout, you randomly turn off nodes during training in order to prevent individual nodes from being too influential, but the nodes are never removed.
  
  You might also be interested in L1 regularization in parameterized models. This is when you add a penalty according to the absolute weights (rather than square weights), which tends to drive less useful weights to 0. Then you can remove the nodes with almost no weight. This is more akin to your decision tree example rather than dropout though.
  
  > **Comments:**
  > **username_2**: It may be also worth mentioning "pruning" in neural networks, as the OP may come across it too.  
  > **username_1**: Ah good point, I forgot about that. Admittedly I don't know much about neural net pruning, I would've guessed it's similar to l1.  
  > **username_3**: xgboost (a decision-tree based algorithm) does use random feature selection and random row selection for different trees, which is more similar to dropout than pruning.  
  > **username_4**: When Random Forest is constructed features of each individual tree are selected at Random, (or in other words dropped out at random), this is done so that the overall model is not too dependent on one feature. This is pretty similar to dropout in Neural Networks.  
  
  ---
  
  