Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

How many trees would you suggest to pick to perform recursive feature elimination (RFE) in order to optimize Random Forest classifier (for binary classification problem). My dataset is very high-dimensional (> 200 000 features), and I usually select ~ 10 000 trees while running a classification without feature selection. But I am just wondering whether it is enough to set it as ~ 500-1000 for RFE in order to save time and RAM.

P.S.: I use 'randomForest' and 'caret' R-packages if it makes any difference.

share|improve this question

migrated from stackoverflow.com Oct 24 '12 at 12:02

This question came from our site for professional and enthusiast programmers.

add comment

2 Answers 2

Optimizing ntree and mtry (above mtry=sqrt(#features) and ntree large enough for stabilization of OOB) is a dangerous area -- you need hard core nested cross-validation to be safe from overfitting, so you may end up doing more computations that you are trying to avoid.

I would say the better idea is not to use RFE -- with 200k features it will have terrible requirements and a minor chance to be stable. Instead, you can use some all-relevant RF wrapper like ACE or Boruta -- the set returned is likely to be larger than minimal-optimal, but still way smaller then the original and thus easier to be treated with RFE.

And remember to validate the feature selection regardless of the method (=

share|improve this answer
add comment

I'm actually doing an experiment with this right now. I work in text classification, so my training set is typically on the order of several hundred thousand features, and I'm looking at comparing a linear SVM (optimized for the c-parameter) against the weka implementation of random forests. I'm finding that, for my data, about 74 trees, and 32 features, thus far, seems to give pretty good performance. Of course, increasing these values tends to increase the AUC I observe, but it's in the thousandths digit place, generally. I'm still trying to understand how this algorithm is handling my data, but I suspect, based on the Breiman paper, that the more generally-useful features there are in your training set, the less important the number of trees parameter becomes. If you read the paper (and it's a really great paper), each tree consists of a random sampling of the features in your data, so, if there are a great many helpful ones in your set, you're more likely to find something useful in any particular tree. That said, I think it's always a good idea to optimize an algorithm for one's particular data. For my experiments, I've set aside a training/optimization set on which I am performing cross-validation across different parameter values. I'd be interested to hear what you find!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.
{"cc_re__cc_re__cc_by_sa_3_0": ["        <li><a href=\"http://area51.stackexchange.com\" title=\"proposing new sites in the Stack Exchange network\">Area 51</a></li>\r\n                        <li><a href=\"http://careers.stackoverflow.com\">Stack Overflow Careers</a></li>\r\n                    \r\n                </ol>\r\n            </td>\r\n    </tr>\r\n</table>\r\n                </div>\r\n            </div>\r\n\r\n            <div id=\"copyright\">\r\n                site design / logo &#169; 2014 stack exchange inc; user contributions licensed under <a href=\"http://creativecommons.org/licenses/by-sa/3.0/\" rel=\"license\">cc by-sa 3.0</a> \r\n                with <a href=\"http://blog.stackoverflow.com/2009/06/attribution-required/\" rel=\"license\">attribution required</a>\r\n            </div>\r\n            <div id=\"svnrev\">\r\n                rev 2014.7.24.1725\r\n            </div>\r\n            \r\n        </div>\r\n    </div>\r\n    <noscript>\r\n        <div id=\"noscript-warning\">Cross Validated works best with JavaScript enabled<img src=\"http://pixel.quantserve.com/pixel/p-c1rF4kxgLUzNc.gif\" alt=\"\" class=\"dno\"></div>\r\n    </no"], "copyright__copyright__copyright": [", I think it's always a good idea to optimize an algorithm for one's particular data. For my experiments, I've set aside a training/optimization set on which I am performing cross-validation across different parameter values. I'd be interested to hear what you find!\n\nshare|improve this answer\nadd comment\n\nYour Answer\n\n\u00a0\ndiscard\n\nBy posting your answer, you agree to the privacy policy and terms of service.\n\nNot the answer you're looking for? Browse other questions tagged or ask your own question."]}