Answers.Body: <p>The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.</p>
  
  <p>As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN.  All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting.  A few can be eliminated up front.  For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.</p>
  
  <p>H. Hutson, S. Geva, and P. Cimiano wrote, in <a href="http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings13/pdf/ntcir/02-NTCIR13-NAILS-HutsonH.pdf" rel="nofollow noreferrer">their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies</a>, "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting."  Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.</p>
  
  <p>Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation."  Reading their paper, <a href="https://arxiv.org/pdf/1702.08835.pdf" rel="nofollow noreferrer">Deep Forest</a>, may give you some causes to evaluate.</p>
  
  <p>Over-fitting is usually the application of too sophisticated a model to which data is fit.  In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.</p>
  
  <p>Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-07T02:33:26.160Z
Answers.LastActivityDate: 2018-07-07T02:33:26.160Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I understand the intuition behind stacking models in machine learning, but even after thorough cross-validation scheme models seem to overfit. Most of the models I have seen in kaggle forums are large ensembles, but seem to overfit very little.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-29T09:21:15.593Z
Comments.Text: Have you tried training each of the stacking models on separate training data?
Comments.UserDisplayName: robit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-29T09:35:18.177Z
Comments.Text: yeah , separate models are perfect , but they overfit a lot when stacked ,
Comments.UserDisplayName: thecomplexitytheorist
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-30T02:14:44.793Z
Comments.Text: It seems weird. I trained a GBDT+LR model: I split training dataset into two subsets A and B. Then, I trained GBDT with A, and LR with B. The stacked model works well. Besides, what models are you training?
Comments.UserDisplayName: robit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-30T05:04:43.780Z
Comments.Text: random forests  and knn
Comments.UserDisplayName: thecomplexitytheorist
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-30T13:24:35.027Z
Comments.Text: @thecomplexitytheorist, please may you describe the CV scheme that you are using. For example, are you using the scheme given [on the Kaggle blog](http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/)? In principle that can lead to overfitting (see the 'Stacked Model Hyper Parameter Tuning' section), though I've never experienced it
Comments.UserDisplayName: Ben
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-30T13:54:16.293Z
Comments.Text: @Ben "But the meta features in {fold2, fold3, fold4, fold5} are dependent on the target values in fold1. So, the target values we’re trying to predict are themselves embedded into the features we’re using to fit our model"  what does that mean?
Comments.UserDisplayName: thecomplexitytheorist
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-04T08:14:21.053Z
Comments.Text: Did you try out any of the regularisation techniques available?
Comments.UserDisplayName: varsh
Tags: <machine-learning>
Title: How to prevent overfitting in stacked models?
UserDisplayName: thecomplexitytheorist
ThreadText: # How to prevent overfitting in stacked models?
  
  (Asked by: username_0 on 2018-06-26)
  
  I understand the intuition behind stacking models in machine learning, but even after thorough cross-validation scheme models seem to overfit. Most of the models I have seen in kaggle forums are large ensembles, but seem to overfit very little.
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: Have you tried training each of the stacking models on separate training data?  
  > **username_0**: yeah , separate models are perfect , but they overfit a lot when stacked ,  
  > **username_1**: It seems weird. I trained a GBDT+LR model: I split training dataset into two subsets A and B. Then, I trained GBDT with A, and LR with B. The stacked model works well. Besides, what models are you training?  
  > **username_0**: random forests  and knn  
  > **username_2**: @thecomplexitytheorist, please may you describe the CV scheme that you are using. For example, are you using the scheme given [on the Kaggle blog](http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/)? In principle that can lead to overfitting (see the 'Stacked Model Hyper Parameter Tuning' section), though I've never experienced it  
  > **username_0**: @Ben "But the meta features in {fold2, fold3, fold4, fold5} are dependent on the target values in fold1. So, the target values we’re trying to predict are themselves embedded into the features we’re using to fit our model"  what does that mean?  
  > **username_3**: Did you try out any of the regularisation techniques available?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2018-07-07. Score: 2
  
  The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.
  
  As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN. All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting. A few can be eliminated up front. For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.
  
  H. Hutson, S. Geva, and P. Cimiano wrote, in [their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies](http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings13/pdf/ntcir/02-NTCIR13-NAILS-HutsonH.pdf), "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting." Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.
  
  Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation." Reading their paper, [Deep Forest](https://arxiv.org/pdf/1702.08835.pdf), may give you some causes to evaluate.
  
  Over-fitting is usually the application of too sophisticated a model to which data is fit. In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.
  
  Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.
  
  
  ---
  
  