<title>Background</title> <p>Gene expression microarrays are becoming increasingly promising for clinical decision support in the form of diagnosis and prediction of clinical outcomes of cancer and other complex diseases. In order to maximize benefits of this technology, researchers are continuously seeking to develop and apply the most accurate classification algorithms for the creation of gene expression patient profiles. Prior research suggests that among well-established and popular techniques for multicategory classification of microarray gene expression data, support vector machines (SVMs) have a predominant role, significantly outperforming k-nearest neighbours, backpropagation neural networks, probabilistic neural networks, weighted voting methods, and decision trees [##REF##15374862##1##].</p> <p>In the last few years substantial interest has developed within the bioinformatics community in the random forest algorithm [##UREF##0##2##] for classification of microarray and other high-dimensional molecular data [##REF##12967959##3##, ####UREF##1##4##, ##REF##16398926##5####16398926##5##]. The random forest algorithm possesses a number of appealing properties making it well-suited for classification of microarray data: (i) it is applicable when there are more predictors than observations, (ii) it performs embedded gene selection and it is relatively insensitive to the large number of irrelevant genes, (iii) it incorporates interactions between predictors, (iv) it is based on the theory of ensemble learning that allows the algorithm to learn accurately both simple and complex classification functions, (v) it is applicable for both binary and multicategory classification tasks, and (vi) according to its inventors it does not require much fine-tuning of parameters and the default parameterization often leads to excellent performance [##UREF##0##2##]. Recent work [##REF##16398926##5##] reported an empirical evaluation of random forests in the cancer microarray gene expression domain and concluded that random forest classifiers have predictive performance comparable to that of the best performing alternatives (including SVMs) for classification of microarray gene expression data. In fact, the data in Table 2 of [##REF##16398926##5##] suggests that random forests on average across 10 datasets slightly outperform SVMs as well as other methods. If true, this finding could be of great significance to the field, because combined with prior results about SVM performance (e.g., [##REF##15374862##1##]), this suggests that random forests offer classification accuracy advantages over "best of class" classifier algorithms for this type of data.</p> <p>However, closer inspection of this prior comparison [##REF##16398926##5##] reveals several important data analytic biases that may have affected its conclusions: First, while the random forests were applied to datasets prior to gene selection, SVMs were applied with a subset of only 200 genes (the number 200 was chosen arbitrarily). Given that the number of optimal genes varies from dataset to dataset, and that SVMs are known to be fairly insensitive to a very large number of irrelevant genes, such application of SVMs likely biases down their performance. Second, a one-versus-one SVM algorithm was applied for the multicategory classification tasks, while it is has been shown that in microarray gene expression domain this method is inferior to other multicategory SVM methods, such as one-versus-rest [##REF##15374862##1##,##UREF##2##6##]. Third, the evaluation of [##REF##16398926##5##] was limited only to linear SVMs without optimizing any algorithm parameters such as the penalty parameter <italic>C </italic>that balances data fit with insensitivity to outliers. Fourth, the performance metric used in [##REF##16398926##5##], proportion of correct classifications, is sensitive to unbalanced distribution of classes and has lower power to discriminate among classification algorithms compared to existing alternatives such as area under the ROC curve and relative classifier information [##UREF##3##7##, ####REF##8668867##8##, ##UREF##4##9##, ##UREF##5##10####5##10##]. Fifth, no statistical comparison among classifiers has been performed. Finally, the prior comparison uses a .632+ bootstrap error estimator [##UREF##6##11##] which is not the most appropriate error estimator for microarray data where powerful classifiers such as SVMs and RFs typically achieve 0 training error and the .632+ bootstrap becomes equivalent to repeated hold-out estimation that may suffer from the training-set-size bias as discussed in [##UREF##7##12##]. Furthermore, .632+ bootstrap is currently not developed for performance metrics other than proportion of correct classifications.</p> <p>We hypothesize that these apparent methodological biases of prior work have compromised its conclusions and the question of whether random forests indeed outperform SVMs for classification of microarray gene expression data is not convincingly answered. In the present work we undertake a more methodologically rigorous comparison of the two algorithms to determine the relative errors when applied to a wide variety of datasets. We examine the algorithms both in the settings when no gene selection is performed and when several popular gene selection methods are used. To make our evaluation more relevant to practitioners, we focus not only on diagnostic datasets that are in general known to have strong predictive signals, but also include several outcome prediction datasets where the signals are weaker and larger gene sets are often required for optimal prediction.</p>
<title>Methods</title> <title>Microarray datasets and classification tasks</title> <p>Gene expression microarray datasets used in the present work are described in Table ##TAB##3##4##. All 22 datasets span the domain of cancer; 14 datasets correspond to diagnostic tasks (and denoted with prefix "Dx") and 8 are concerned with clinical outcome prediction (and denoted with "Px"). Out of 22 datasets, 11 are binary classification tasks, while the other 11 are multicategory tasks with 3–26 classes. The datasets contain 50–308 samples and 2,000–24,188 variables (genes) after data preparatory steps described in [##REF##15374862##1##]. All diagnostic datasets were obtained from <ext-link ext-link-type="uri" xlink:href="http://www.gems-system.org"/>[##REF##15374862##1##] and from the links given in the primary study for each dataset. Similarly, all prognostic datasets were obtained from the links given in the primary study for each dataset. A list of references to the primary study for each dataset is provided in the Additional File ##SUPPL##2##3##. Notice that the dataset collection used in this work contains all datasets from the prior comparison [##REF##16398926##5##].</p> <title>Cross-validation design</title> <p>We used 10-fold cross-validation to estimate the performance of the classification algorithms. In order to optimize algorithm parameters, we used another "nested" loop of cross-validation by further splitting each of the 10 original training sets into smaller training sets and validation sets. For each combination of the classifier parameters, we obtained cross-validation performance and selected the best performing parameters inside this inner loop of cross-validation. Next, we built a classification model with the best parameters on the original training set and applied this model to the original testing set. Details about the "nested cross-validation" procedure can be found in [##REF##15967710##19##,##UREF##12##20##]. Notice that the final performance estimate obtained by this procedure will be unbiased because each original testing set is used only once to estimate performance of a single classification model that was built by using training data exclusively.</p> <title>Support vector machine classifiers</title> <p>Several theoretical reasons explain the superior empirical performance of SVMs in microarray data: e.g., they are robust to the high variable-to-sample ratio and large number of variables, they can learn efficiently complex classification functions, and they employ powerful regularization principles to avoid overfitting [##REF##15374862##1##,##REF##11120680##21##,##UREF##13##22##]. Extensive applications literature in text categorization, image recognition and other fields also shows the excellent empirical performance of this classifier in many more domains. The underlying idea of SVM classifiers is to calculate a maximal margin hyperplane separating two classes of the data. To learn non-linearly separable functions, the data are implicitly mapped to a higher dimensional space by means of a kernel function, where a separating hyperplane is found. New samples are classified according to the side of the hyperplane they belong to [##UREF##13##22##]. Many extensions of the basic SVM algorithm can handle multicategory data. The "one-versus-rest" SVM works better for multi-class microarray data [##REF##15374862##1##,##UREF##2##6##], so we adopted this method for the analysis of multicategory datasets in the present study. In summary, this approach involves building a separate SVM model to classify each class against the rest, and then predicting the class of a new sample using the SVM model with the strongest vote.</p> <p>We used SVM implementation in the <italic>libSVM </italic>software library [##UREF##14##23##]<ext-link ext-link-type="uri" xlink:href="http://www.csie.ntu.edu.tw/~cjlin/libsvm"/> with polynomial kernel. Recall that the SVM polynomial kernel can be defined as: <italic>K</italic>(<italic>x</italic>, <italic>y</italic>) = (γ·<italic>x</italic><sup>T</sup><italic>y </italic>+ <italic>r</italic>)<sup><italic>d</italic></sup>, where <italic>x </italic>and <italic>y </italic>are samples with gene expression values and γ, <italic>r</italic>, <italic>d </italic>are kernel parameters. The parameters γ and <italic>r </italic>were set to default value 1. The kernel degree <italic>d </italic>together with the SVM penalty parameter <italic>C </italic>were optimized by nested cross-validation over <italic>d </italic>values {1, 2, 3} and <italic>C </italic>values {0.01, 1, 100}.</p> <title>Random forest classifiers</title> <p>Random forests (RF) is a classification algorithm that uses an ensemble of unpruned decision trees, each of which is built on a bootstrap sample of the training data using a randomly selected subset of variables [##UREF##0##2##]. As mentioned in the Background section, this algorithm possesses a number of properties making it an attractive technique for classification of microarray gene expression data.</p> <p>We employed the state-of-the-art implementation of RF available in the R package <italic>randomForest </italic>[##UREF##15##24##]. This implementation is based on the original Fortran code authored by Leo Breiman, the inventor of RFs. Following the suggestions of [##UREF##15##24##,##UREF##16##25##] and <ext-link ext-link-type="uri" xlink:href="http://www.stat.berkeley.edu/~breiman/RandomForests/"/>, we considered different parameter configurations for the values of <italic>ntree </italic>= {500, 1000, 2000} (number of trees to build), <italic>mtryFactor </italic>={0.5, 1, 2} (a multiplicative factor of the default value of <italic>mtry </italic>parameter denoting the number of genes randomly selected at each node; by default <italic>mtry </italic>= ), and <italic>nodesize </italic>= 1 (minimal size of the terminal nodes of the trees in a random forest) and selected the best-performing configuration by nested cross-validation. Note that the above parameter values are also consistent with the recommendations of the study [##REF##16398926##5##].</p> <title>Gene selection methods</title> <p>Even though both SVM and RF classifiers are fairly insensitive to very large number of irrelevant genes, we applied the following widely used gene selection methods in order to further improve classification performance:</p> <p>• Random forest-based backward elimination procedure RFVS [##REF##16398926##5##]: The RFVS procedure involves iteratively fitting RFs (on the training data), and at each iteration building a random forest after discarding genes with the smallest importance values. The returned subset of genes is the one with the smallest out-of-bag error. We used the <italic>varSelRF </italic>implementation of the RFVS method developed by its inventors and applied it with the recommended parameters: <italic>ntree </italic>= 2000, <italic>mtryFactor </italic>= 1, <italic>nodesize </italic>= 1, <italic>fraction.dropped </italic>= 0.2 (a parameter denoting fraction of genes with small importance values to be discarded during backward elimination procedure), and <italic>c.sd </italic>= 0 (a factor that multiplies the standard deviation of error for stopping iterations and choosing the best performing subset of genes). We refer to this method as "RFVS1."</p> <p>• RFVS procedure as described above, except for <italic>c.sd </italic>= 1 (denoted as "RFVS2"): This method differs from RFVS1 in that it performs statistical comparison to return the smallest subset of genes with performance statistically indistinguishable from the nominally best one.</p> <p>• SVM-based recursive feature elimination method RFE [##UREF##17##26##]: This is a state-of-the-art procedure for gene selection from microarray data that involves iteratively fitting SVM classification models (on the training data) by discarding the genes with the small impact on classification and selecting the smallest subset of genes that participate in the best performing classification model (as assessed in the validation data). Even though RFE was originally introduced as a method for binary classification problems, it can be trivially extended to multiclass case by using binary SVM models in "one-versus-rest" fashion (e.g., see [##UREF##18##27##]). Finally, to be comparable with the RFVS method, we used the fraction of genes that are discarded in the iterative SVM models equal to 0.2.</p> <p>• Backward elimination procedure based on univariate ranking of genes with "signal-to-noise" ratio [##REF##15374862##1##,##REF##11120680##21##,##REF##10521349##28##] (denoted as "S2N"): This procedure first ranks all genes according their signal-to-noise value with the response variable, and then performs backward elimination using SVM classifier (fit on the training set and evaluated on the validation set) to determine the best performing smallest subset of genes. Similarly to RFE and RFVS, we perform backward elimination by discarding 0.2 proportion of genes at each iteration.</p> <p>• Backward elimination procedure based on univariate ranking of genes with Kruskal-Wallis one-way non-parametric ANOVA [##REF##15374862##1##] (denoted as "KW"): This procedure is applied similarly to the S2N method except for it uses different univariate ranking of genes.</p> <p>We emphasize that all gene selection methods were applied during cross-validation utilizing only the training data and splitting it into a smaller training and validation set if necessary.</p> <title>Classification performance evaluation metrics</title> <p>We used two classification performance metrics. For binary tasks, we used the area under the ROC curve (AUC) which was computed from continuous outputs of the classifiers (distances from separating hyperplane for SVMs and outcome probabilities for RFs) [##REF##8668867##8##]. For multicategory tasks, where classical AUC is inapplicable, we employed the relative classifier information (RCI) [##UREF##3##7##]. RCI is an entropy-based measure that quantifies how much the uncertainty of a decision problem is reduced by a classifier relative to classifying using only the prior probabilities of each class. We note that both AUC and RCI are more discriminative than the accuracy metric (also known as proportion of correct classifications) and are not sensitive to unbalanced distributions [##UREF##3##7##, ####REF##8668867##8##, ##UREF##4##9##, ##UREF##5##10####5##10##]. Both AUC and RCI take values on [0, 1], where 0 denotes worst possible classification and 1 denotes perfect classification.</p> <title>Statistical comparison among classifiers</title> <p>When comparing two classifiers, it is important to assess whether the observed difference in classification performance is statistically significant or simply due to chance. We assessed significance of differences in classification performance in individual datasets or in all datasets on average using a non-parametric permutation test [##UREF##19##29##] based on the theory of [##UREF##20##30##]. The null hypothesis of this test is no difference between performance of SVM and RF classifiers. The test was applied with 100,000 permutations and two-sided p-values were computed as described in [##UREF##19##29##]. We used a significance level α = 0.05 for this test.</p>
<title>Results</title> <title>Using full set of genes</title> <p>The performance results of classification prior to gene selection are shown in Figure ##FIG##0##1## and Table ##TAB##0##1##. In total, SVMs nominally (that is, not necessarily statistically significantly) outperform RFs in 15 datasets, RFs nominally outperform SVMs in 4 datasets, and in 3 datasets algorithms perform the same. The application of permutation-based statistical comparison test with significance level α = 0.05 reveals that SVMs significantly outperform RFs in 7 datasets, while RFs do not significantly outperform SVMs in any dataset. The permutation test applied to all 22 datasets shows that SVMs statistically significantly outperform RFs on average over all datasets at the 0.05 α level (p-value of the test = 0.008). It is also worthwhile to compare both methods in terms of the average performance across datasets. The average performance of SVMs is 0.775 AUC and 0.860 RCI in binary and multicategory classification tasks, respectively. The average performance of RFs in the same tasks is 0.742 AUC and 0.803 RCI.</p> <title>Using gene selection</title> <p>Six classification performance estimates have been produced for each classifier and dataset (5 estimates corresponding to various gene selection methods and one estimate corresponding to using no gene selection). In Figure ##FIG##1##2## and Table ##TAB##1##2## we present a comparison based on the best performing gene selection method for each algorithm and dataset combination under the operating assumption that practitioners will optimize choice of the gene selection method for each dataset separately (using cross-validation or other suitable protocols). The results in Figure ##FIG##1##2## and Table ##TAB##1##2## thus better mirror the actual practice of data analysis.</p> <p>According to the results in Figure ##FIG##1##2## and Table ##TAB##1##2##, in 17 datasets SVMs nominally outperform RFs, in 3 datasets RFs nominally outperform SVMs, and in 2 datasets algorithms perform the same. Furthermore, SVMs outperform RFs statistically significantly (at the 0.05 α level) in 1 dataset. There is no dataset where RFs outperform SVMs with statistically significant difference. The permutation test applied to all 22 datasets shows that SVMs statistically significantly outperform RFs on average over all datasets at the 0.05 α level (p-value of the test = 0.001). A comparison of the average performance across datasets also suggests superiority of SVMs: the average performance of SVMs is 0.787 AUC and 0.875 RCI in binary and multicategory classification tasks, respectively; while the average performance of RFs in the same tasks is 0.759 AUC and 0.828 RCI.</p> <p>The number of genes selected on average across 10 cross-validation training sets is provided in Table ##TAB##2##3##. We note that in the present comparison we focus exclusively on classification performance and do not incorporate number of selected genes in the comparison metrics because there is no well-defined trade-off between number of selected genes and classification performance in the datasets studied. Nevertheless, the detailed classification results for all gene selection methods, classifiers, and datasets are provided in the Additional File ##SUPPL##0##1##.</p>
<title>Discussion</title> <p>The results presented in this paper illustrate that SVMs offer classification performance advantages compared to RFs in diagnostic and prognostic classification tasks based on microarray gene expression data. We emphasize that when it comes to clinical applications of such models, because the size of the patient populations is typically very large, even very modest differences in performance (e.g., at the order of 0.01 AUC/RCI or even less) can result in very substantial differences in total clinical outcomes (e.g., number of life-years saved) [##REF##17074082##13##].</p> <p>The reasons for superior classification performance of one universal approximator classifier over the other in a domain where the generative functions are unknown are not trivial to decipher [##UREF##0##2##,##UREF##8##14##]. We provide here as a starting point two plausible explanations supported by theory and a simulation experiment (in Additional File ##SUPPL##1##2##). We note that prior research has established that linear decision functions capture very well the underlying distributions in microarray classification tasks [##UREF##9##15##,##REF##17227998##16##]. In the following two paragraphs we first demonstrate that for such functions SVMs may be less sensitive to the choice of input parameters than RFs and then explain why SVMs model linear decision functions more naturally than RFs.</p> <p>The simulation experiment described in Additional File ##SUPPL##1##2## demonstrates high degree of sensitivity of RFs to the values of input parameters <italic>mtry </italic>(i.e., number of genes randomly selected at each node) and <italic>ntree </italic>(i.e., number of trees) even in the case of linear decision function when complicated decision surface modelling is not required. The experiment shows that the choice of RF parameters creates large variation in the classifier performance whereas the choice of the main SVM parameter has only minor effects on the error. In practical analysis of microarrays this means that finding the RFs with optimal error for the dataset may involve extensive model selection which in turn opens up the possibility for overfitting given the small sample sizes in validation datasets.</p> <p>A second plausible explanation is that decision trees used as base learners in the RF algorithm cannot learn exactly many linear decision functions in the finite case. Specifically, if the generative linear decision function is not orthogonal to the coordinate axes, then a decision tree of infinite size is required to represent this function without error [##UREF##10##17##]. The voted decision function in RFs approximates linear functions based on rectangular partitioning of the input space, and this "staircase" approximation can capture a linear function exactly when the number of decision trees can grow without bound (assuming that each tree is of finite size). SVMs on the other hand use linear classifiers and thus can model such functions naturally, using a small number of free parameters (i.e., bounded by the available sample size).</p> <p>We note that regardless of the specific reasons why RFs may have larger error on average in this domain, it is still important to be aware of the empirical performance differences when considering which classifier to use for building molecular signatures. It may take several years before the precise reasons of differences in empirical error are thoroughly understood, and in the meantime the empirical advantages and disadvantages of methods should be noted first by practitioners.</p> <p>Data analysts should also be aware of a limitation of RFs imposed by its embedded random gene selection. In order for a RF classification model to overcome the trap of large variance, one has to use a large number of trees and build trees based on a large number of genes. The exact values of these parameters depend on both the complexity of the classification function and the number of genes in a microarray dataset. Therefore, in general, it is advisable to optimize these parameters by nested cross-validation that accounts for the variability of the random forest model (e.g., the selected parameter configuration is the one that performs best on average over multiple validation sample sets).</p> <p>Finally, it is worthwhile to mention the work by Segal [##UREF##11##18##] who questioned Breiman's empirical demonstration of the claim that random forests do not overfit as the number of trees grows [##UREF##0##2##]. In short, Segal showed that there exist some data distributions where maximal unpruned trees used in the random forests do not achieve as good performance as the trees with smaller number of splits and/or smaller node size. Thus, application of random forests in general requires careful tuning of the relevant classifier parameters. These observations may suggest future improvements of RF-related analysis protocols.</p>
<title>Conclusion</title> <p>The primary contribution of the present work is that we conducted the most comprehensive comparative benchmarking of random forests and support vector machines to date, using 22 diagnostic and outcome prediction datasets. Our hypothesis that in previously reported work, research design limitations may have biased the comparison of classifiers in favour of random forests, was verified. After removing these benchmarking limitations, we found that, both on average and in the majority of microarray datasets, random forests exhibit larger classification error than support vector machines both in the settings when no gene selection is performed and when several gene selection methods are used.</p> <p>The quest for high performance classifiers with microarray gene expression and other "omics" data is ongoing. Random forests have appealing theoretical and practical characteristics, however our experiments show that currently they do not exhibit "best of class" performance. Our data also points to methodological limitations of prior evaluations and thus emphasizes the importance of careful design of bioinformatics algorithm evaluation studies.</p>
<p>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/2.0"/>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</p> <title>Background</title> <p>Cancer diagnosis and clinical outcome prediction are among the most important emerging applications of gene expression microarray technology with several molecular signatures on their way toward clinical deployment. Use of the most accurate classification algorithms available for microarray gene expression data is a critical ingredient in order to develop the best possible molecular signatures for patient care. As suggested by a large body of literature to date, support vector machines can be considered "best of class" algorithms for classification of such data. Recent work, however, suggests that random forest classifiers may outperform support vector machines in this domain.</p> <title>Results</title> <p>In the present paper we identify methodological biases of prior work comparing random forests and support vector machines and conduct a new rigorous evaluation of the two algorithms that corrects these limitations. Our experiments use 22 diagnostic and prognostic datasets and show that support vector machines outperform random forests, often by a large margin. Our data also underlines the importance of sound research design in benchmarking and comparison of bioinformatics algorithms.</p> <title>Conclusion</title> <p>We found that both on average and in the majority of microarray datasets, random forests are outperformed by support vector machines both in the settings when no gene selection is performed and when several popular gene selection methods are used.</p>
<title>Authors' contributions</title> <p>Conceived and designed the experiments: AS, LW, CFA. Performed the experiments: AS. Analyzed the results of experiments: AS, LW, CFA. Wrote the paper: AS, CFA. All authors read and approved the final manuscript.</p> <title>Supplementary Material</title>
<title>Acknowledgements</title> <p>The work was in part supported by grant 2R56LM007948-04A1.</p>
<fig position="float" id="F1"><label>Figure 1</label><caption><p><bold>Classification performance of SVMs and RFs without gene selection</bold>. The performance is estimated using area under ROC curve (AUC) for binary classification tasks and relative classifier information (RCI) for multicategory tasks.</p></caption></fig> <fig position="float" id="F2"><label>Figure 2</label><caption><p><bold>Classification performance of SVMs and RFs with gene selection</bold>. The performance is estimated using area under ROC curve (AUC) for binary classification tasks and relative classifier information (RCI) for multicategory tasks.</p></caption></fig>
<table-wrap position="float" id="T1"><label>Table 1</label><caption><p>Comparison of classification performance of SVMs and RFs without gene selection.</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="left"><bold><italic>Task &amp; dataset</italic></bold></td><td align="center"><bold><italic>Classification performance metric</italic></bold></td><td align="center" colspan="2"><bold><italic>Classification performance</italic></bold></td><td align="center"><bold><italic>Nominally superior method</italic></bold></td><td align="left"><bold><italic>P-value</italic></bold></td></tr><tr><td/><td/><td colspan="2"><hr/></td><td/><td/></tr><tr><td/><td/><td align="center"><bold><italic>SVM</italic></bold></td><td align="center"><bold><italic>RF</italic></bold></td><td/><td/></tr></thead><tbody><tr><td align="left"><italic>Dx-Alon</italic></td><td align="center">AUC</td><td align="center">0.867</td><td align="center">0.867</td><td align="center">-</td><td align="left">1</td></tr><tr><td align="left"><italic>Dx-Ramaswamy2</italic></td><td align="center">AUC</td><td align="center">0.821</td><td align="center">0.767</td><td align="center">SVM</td><td align="left">0.409</td></tr><tr><td align="left"><italic>Dx-Shipp</italic></td><td align="center">AUC</td><td align="center">0.992</td><td align="center">0.973</td><td align="center">SVM</td><td align="left">0.500</td></tr><tr><td align="left"><italic>Dx-Singh</italic></td><td align="center">AUC</td><td align="center">0.964</td><td align="center">0.944</td><td align="center">SVM</td><td align="left">0.377</td></tr><tr><td align="left"><italic>Px-Beer</italic></td><td align="center">AUC</td><td align="center">0.798</td><td align="center">0.646</td><td align="center">SVM</td><td align="left"><bold>0.032</bold></td></tr><tr><td align="left"><italic>Px-Bhattacharjee</italic></td><td align="center">AUC</td><td align="center">0.519</td><td align="center">0.561</td><td align="center">RF</td><td align="left">0.546</td></tr><tr><td align="left"><italic>Px-Iizuka</italic></td><td align="center">AUC</td><td align="center">0.663</td><td align="center">0.763</td><td align="center">RF</td><td align="left">0.061</td></tr><tr><td align="left"><italic>Px-Pomeroy</italic></td><td align="center">AUC</td><td align="center">0.692</td><td align="center">0.600</td><td align="center">SVM</td><td align="left">0.235</td></tr><tr><td align="left"><italic>Px-Rosenwald</italic></td><td align="center">AUC</td><td align="center">0.689</td><td align="center">0.629</td><td align="center">SVM</td><td align="left">0.140</td></tr><tr><td align="left"><italic>Px-Veer</italic></td><td align="center">AUC</td><td align="center">0.747</td><td align="center">0.754</td><td align="center">RF</td><td align="left">0.867</td></tr><tr><td align="left"><italic>Px-Yeoh</italic></td><td align="center">AUC</td><td align="center">0.777</td><td align="center">0.660</td><td align="center">SVM</td><td align="left"><bold>0.006</bold></td></tr><tr><td colspan="6"><hr/></td></tr><tr><td align="left"><italic>Dx-Alizadeh</italic></td><td align="center">RCI</td><td align="center">1.000</td><td align="center">1.000</td><td align="center">-</td><td align="left">1</td></tr><tr><td align="left"><italic>Dx-Armstrong</italic></td><td align="center">RCI</td><td align="center">0.944</td><td align="center">0.894</td><td align="center">SVM</td><td align="left">0.658</td></tr><tr><td align="left"><italic>Dx-Bhattacharjee</italic></td><td align="center">RCI</td><td align="center">0.895</td><td align="center">0.763</td><td align="center">SVM</td><td align="left"><bold>0.015</bold></td></tr><tr><td align="left"><italic>Dx-Golub</italic></td><td align="center">RCI</td><td align="center">0.939</td><td align="center">0.934</td><td align="center">SVM</td><td align="left">1</td></tr><tr><td align="left"><italic>Dx-Khan</italic></td><td align="center">RCI</td><td align="center">1.000</td><td align="center">1.000</td><td align="center">-</td><td align="left">1</td></tr><tr><td align="left"><italic>Dx-Nutt</italic></td><td align="center">RCI</td><td align="center">0.775</td><td align="center">0.733</td><td align="center">SVM</td><td align="left">0.498</td></tr><tr><td align="left"><italic>Dx-Pomeroy</italic></td><td align="center">RCI</td><td align="center">0.823</td><td align="center">0.611</td><td align="center">SVM</td><td align="left"><bold>0.031</bold></td></tr><tr><td align="left"><italic>Dx-Ramaswamy</italic></td><td align="center">RCI</td><td align="center">0.905</td><td align="center">0.861</td><td align="center">SVM</td><td align="left"><bold>0.010</bold></td></tr><tr><td align="left"><italic>Dx-Staunton</italic></td><td align="center">RCI</td><td align="center">0.770</td><td align="center">0.819</td><td align="center">RF</td><td align="left">0.249</td></tr><tr><td align="left"><italic>Dx-Su</italic></td><td align="center">RCI</td><td align="center">0.958</td><td align="center">0.910</td><td align="center">SVM</td><td align="left"><bold>0.004</bold></td></tr><tr><td align="left"><italic>Px-Veer2</italic></td><td align="center">RCI</td><td align="center">0.451</td><td align="center">0.304</td><td align="center">SVM</td><td align="left"><bold>0.004</bold></td></tr></tbody></table></table-wrap> <table-wrap position="float" id="T2"><label>Table 2</label><caption><p>Comparison of classification performance of SVMs and RFs with gene selection.</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="left"><bold><italic>Task &amp; dataset</italic></bold></td><td align="center"><bold><italic>Classification performance metric</italic></bold></td><td align="center" colspan="2"><bold><italic>Classification performance</italic></bold></td><td align="center"><bold><italic>Nominally superior method</italic></bold></td><td align="center"><bold><italic>P-value</italic></bold></td></tr><tr><td/><td/><td colspan="2"><hr/></td><td/><td/></tr><tr><td/><td/><td align="center"><bold><italic>SVM</italic></bold></td><td align="center"><bold><italic>RF</italic></bold></td><td/><td/></tr></thead><tbody><tr><td align="left"><italic>Dx-Alon</italic></td><td align="center">AUC</td><td align="center">0.938</td><td align="center">0.917</td><td align="center">SVM</td><td align="center">0.626</td></tr><tr><td align="left"><italic>Dx-Ramaswamy2</italic></td><td align="center">AUC</td><td align="center">0.821</td><td align="center">0.781</td><td align="center">SVM</td><td align="center">0.624</td></tr><tr><td align="left"><italic>Dx-Shipp</italic></td><td align="center">AUC</td><td align="center">0.992</td><td align="center">0.975</td><td align="center">SVM</td><td align="center">0.502</td></tr><tr><td align="left"><italic>Dx-Singh</italic></td><td align="center">AUC</td><td align="center">0.964</td><td align="center">0.972</td><td align="center">RF</td><td align="center">0.812</td></tr><tr><td align="left"><italic>Px-Beer</italic></td><td align="center">AUC</td><td align="center">0.798</td><td align="center">0.648</td><td align="center">SVM</td><td align="center"><bold>0.016</bold></td></tr><tr><td align="left"><italic>Px-Bhattacharjee</italic></td><td align="center">AUC</td><td align="center">0.519</td><td align="center">0.561</td><td align="center">RF</td><td align="center">0.550</td></tr><tr><td align="left"><italic>Px-Iizuka</italic></td><td align="center">AUC</td><td align="center">0.713</td><td align="center">0.763</td><td align="center">RF</td><td align="center">0.750</td></tr><tr><td align="left"><italic>Px-Pomeroy</italic></td><td align="center">AUC</td><td align="center">0.692</td><td align="center">0.629</td><td align="center">SVM</td><td align="center">0.506</td></tr><tr><td align="left"><italic>Px-Rosenwald</italic></td><td align="center">AUC</td><td align="center">0.689</td><td align="center">0.631</td><td align="center">SVM</td><td align="center">0.128</td></tr><tr><td align="left"><italic>Px-Veer</italic></td><td align="center">AUC</td><td align="center">0.758</td><td align="center">0.754</td><td align="center">SVM</td><td align="center">0.954</td></tr><tr><td align="left"><italic>Px-Yeoh</italic></td><td align="center">AUC</td><td align="center">0.777</td><td align="center">0.716</td><td align="center">SVM</td><td align="center">0.082</td></tr><tr><td colspan="6"><hr/></td></tr><tr><td align="left"><italic>Dx-Alizadeh</italic></td><td align="center">RCI</td><td align="center">1.000</td><td align="center">1.000</td><td align="center">-</td><td align="center">1</td></tr><tr><td align="left"><italic>Dx-Armstrong</italic></td><td align="center">RCI</td><td align="center">0.944</td><td align="center">0.911</td><td align="center">SVM</td><td align="center">0.624</td></tr><tr><td align="left"><italic>Dx-Bhattacharjee</italic></td><td align="center">RCI</td><td align="center">0.895</td><td align="center">0.817</td><td align="center">SVM</td><td align="center">0.125</td></tr><tr><td align="left"><italic>Dx-Golub</italic></td><td align="center">RCI</td><td align="center">0.953</td><td align="center">0.934</td><td align="center">SVM</td><td align="center">1</td></tr><tr><td align="left"><italic>Dx-Khan</italic></td><td align="center">RCI</td><td align="center">1.000</td><td align="center">1.000</td><td align="center">-</td><td align="center">1</td></tr><tr><td align="left"><italic>Dx-Nutt</italic></td><td align="center">RCI</td><td align="center">0.812</td><td align="center">0.733</td><td align="center">SVM</td><td align="center">0.220</td></tr><tr><td align="left"><italic>Dx-Pomeroy</italic></td><td align="center">RCI</td><td align="center">0.823</td><td align="center">0.688</td><td align="center">SVM</td><td align="center">0.079</td></tr><tr><td align="left"><italic>Dx-Ramaswamy</italic></td><td align="center">RCI</td><td align="center">0.911</td><td align="center">0.880</td><td align="center">SVM</td><td align="center">0.066</td></tr><tr><td align="left"><italic>Dx-Staunton</italic></td><td align="center">RCI</td><td align="center">0.876</td><td align="center">0.856</td><td align="center">SVM</td><td align="center">0.626</td></tr><tr><td align="left"><italic>Dx-Su</italic></td><td align="center">RCI</td><td align="center">0.958</td><td align="center">0.922</td><td align="center">SVM</td><td align="center">0.078</td></tr><tr><td align="left"><italic>Px-Veer2</italic></td><td align="center">RCI</td><td align="center">0.451</td><td align="center">0.371</td><td align="center">SVM</td><td align="center">0.262</td></tr></tbody></table></table-wrap> <table-wrap position="float" id="T3"><label>Table 3</label><caption><p>Number of genes selected for each microarray dataset and gene selection method.</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="left"><bold><italic>Task &amp; dataset</italic></bold></td><td align="center"><bold><italic>No gene selection</italic></bold></td><td align="center"><bold><italic>RFE</italic></bold></td><td align="center"><bold><italic>RFVS1</italic></bold></td><td align="center"><bold><italic>RFVS2</italic></bold></td><td align="center"><bold><italic>KW</italic></bold></td><td align="center"><bold><italic>S2N</italic></bold></td></tr></thead><tbody><tr><td align="left"><italic>Dx-Alizadeh</italic></td><td align="center">4026</td><td align="center">12</td><td align="center">62</td><td align="center">73</td><td align="center">19</td><td align="center">15</td></tr><tr><td align="left"><italic>Dx-Alon</italic></td><td align="center">2000</td><td align="center">105</td><td align="center">16</td><td align="center">3</td><td align="center">15</td><td align="center">13</td></tr><tr><td align="left"><italic>Dx-Armstrong</italic></td><td align="center">11225</td><td align="center">74</td><td align="center">709</td><td align="center">57</td><td align="center">106</td><td align="center">48</td></tr><tr><td align="left"><italic>Dx-Bhattacharjee</italic></td><td align="center">12600</td><td align="center">289</td><td align="center">27</td><td align="center">15</td><td align="center">1864</td><td align="center">653</td></tr><tr><td align="left"><italic>Dx-Golub</italic></td><td align="center">5327</td><td align="center">12</td><td align="center">456</td><td align="center">336</td><td align="center">42</td><td align="center">4</td></tr><tr><td align="left"><italic>Dx-Khan</italic></td><td align="center">2308</td><td align="center">28</td><td align="center">17</td><td align="center">18</td><td align="center">15</td><td align="center">11</td></tr><tr><td align="left"><italic>Dx-Nutt</italic></td><td align="center">10367</td><td align="center">1598</td><td align="center">126</td><td align="center">101</td><td align="center">476</td><td align="center">926</td></tr><tr><td align="left"><italic>Dx-Pomeroy</italic></td><td align="center">5920</td><td align="center">186</td><td align="center">34</td><td align="center">16</td><td align="center">70</td><td align="center">435</td></tr><tr><td align="left"><italic>Dx-Ramaswamy</italic></td><td align="center">15009</td><td align="center">3346</td><td align="center">966</td><td align="center">411</td><td align="center">8248</td><td align="center">10277</td></tr><tr><td align="left"><italic>Dx-Ramaswamy2</italic></td><td align="center">13247</td><td align="center">1576</td><td align="center">12</td><td align="center">4</td><td align="center">4129</td><td align="center">1364</td></tr><tr><td align="left"><italic>Dx-Shipp</italic></td><td align="center">5469</td><td align="center">8</td><td align="center">15</td><td align="center">6</td><td align="center">13</td><td align="center">89</td></tr><tr><td align="left"><italic>Dx-Singh</italic></td><td align="center">10509</td><td align="center">157</td><td align="center">58</td><td align="center">21</td><td align="center">22</td><td align="center">38</td></tr><tr><td align="left"><italic>Dx-Staunton</italic></td><td align="center">5726</td><td align="center">169</td><td align="center">152</td><td align="center">73</td><td align="center">93</td><td align="center">97</td></tr><tr><td align="left"><italic>Dx-Su</italic></td><td align="center">12533</td><td align="center">2429</td><td align="center">845</td><td align="center">320</td><td align="center">1318</td><td align="center">1927</td></tr><tr><td align="left"><italic>Px-Beer</italic></td><td align="center">7129</td><td align="center">201</td><td align="center">15</td><td align="center">7</td><td align="center">953</td><td align="center">1380</td></tr><tr><td align="left"><italic>Px-Bhattacharjee</italic></td><td align="center">12600</td><td align="center">21</td><td align="center">46</td><td align="center">7</td><td align="center">138</td><td align="center">61</td></tr><tr><td align="left"><italic>Px-Iizuka</italic></td><td align="center">7070</td><td align="center">103</td><td align="center">38</td><td align="center">7</td><td align="center">168</td><td align="center">185</td></tr><tr><td align="left"><italic>Px-Pomeroy</italic></td><td align="center">7129</td><td align="center">70</td><td align="center">29</td><td align="center">13</td><td align="center">445</td><td align="center">439</td></tr><tr><td align="left"><italic>Px-Rosenwald</italic></td><td align="center">7399</td><td align="center">2338</td><td align="center">124</td><td align="center">27</td><td align="center">3201</td><td align="center">3897</td></tr><tr><td align="left"><italic>Px-Veer</italic></td><td align="center">24188</td><td align="center">1056</td><td align="center">124</td><td align="center">20</td><td align="center">5388</td><td align="center">4405</td></tr><tr><td align="left"><italic>Px-Veer2</italic></td><td align="center">24188</td><td align="center">491</td><td align="center">149</td><td align="center">39</td><td align="center">1194</td><td align="center">1764</td></tr><tr><td align="left"><italic>Px-Yeoh</italic></td><td align="center">12240</td><td align="center">1187</td><td align="center">21</td><td align="center">6</td><td align="center">3077</td><td align="center">1869</td></tr></tbody></table></table-wrap> <table-wrap position="float" id="T4"><label>Table 4</label><caption><p>Gene expression microarray datasets used in this study.</p></caption><table frame="hsides" rules="groups"><thead><tr><td align="left"><bold><italic>Task &amp; dataset</italic></bold></td><td align="center"><bold><italic>Number of classes</italic></bold></td><td align="center"><bold><italic>Number of genes</italic></bold></td><td align="center"><bold><italic>Number of samples</italic></bold></td><td align="left"><bold><italic>Prediction task</italic></bold></td></tr></thead><tbody><tr><td align="left"><italic>Dx-Alizadeh</italic></td><td align="center">3</td><td align="center">4026</td><td align="center">62</td><td align="left">Diffuse large B-cell lymphoma, follicular lymphoma, chronic lymphocytic leukemia</td></tr><tr><td align="left"><italic>Dx-Alon</italic></td><td align="center">2</td><td align="center">2000</td><td align="center">62</td><td align="left">Colon tumors and normal tissues</td></tr><tr><td align="left"><italic>Dx-Armstrong</italic></td><td align="center">3</td><td align="center">11225</td><td align="center">72</td><td align="left">AML, ALL and mixed-lineage leukemia (MLL)</td></tr><tr><td align="left"><italic>Dx-Bhattacharjee</italic></td><td align="center">5</td><td align="center">12600</td><td align="center">203</td><td align="left">4 lung cancer types and normal tissues</td></tr><tr><td align="left"><italic>Dx-Golub</italic></td><td align="center">3</td><td align="center">5327</td><td align="center">72</td><td align="left">Acute myelogenous leukemia (AML), acute lymphoblastic leukemia (ALL) B-cell and ALL T-cell</td></tr><tr><td align="left"><italic>Dx-Khan</italic></td><td align="center">4</td><td align="center">2308</td><td align="center">83</td><td align="left">Small, round blue cell tumors of childhood</td></tr><tr><td align="left"><italic>Dx-Nutt</italic></td><td align="center">4</td><td align="center">10367</td><td align="center">50</td><td align="left">4 malignant glioma types</td></tr><tr><td align="left"><italic>Dx-Pomeroy</italic></td><td align="center">5</td><td align="center">5920</td><td align="center">90</td><td align="left">5 human brain tumor types</td></tr><tr><td align="left"><italic>Dx-Ramaswamy</italic></td><td align="center">26</td><td align="center">15009</td><td align="center">308</td><td align="left">14 various human tumor types and 12 normal tissue types</td></tr><tr><td align="left"><italic>Dx-Ramaswamy2</italic></td><td align="center">2</td><td align="center">13247</td><td align="center">76</td><td align="left">Metastatic and primary tumors</td></tr><tr><td align="left"><italic>Dx-Shipp</italic></td><td align="center">2</td><td align="center">5469</td><td align="center">77</td><td align="left">Diffuse large B-cell lymphomas and follicular lymphomas</td></tr><tr><td align="left"><italic>Dx-Singh</italic></td><td align="center">2</td><td align="center">10509</td><td align="center">102</td><td align="left">Prostate tumor and normal tissues</td></tr><tr><td align="left"><italic>Dx-Staunton</italic></td><td align="center">9</td><td align="center">5726</td><td align="center">60</td><td align="left">9 various human tumor types</td></tr><tr><td align="left"><italic>Dx-Su</italic></td><td align="center">11</td><td align="center">12533</td><td align="center">174</td><td align="left">11 various human tumor types</td></tr><tr><td align="left"><italic>Px-Beer</italic></td><td align="center">2</td><td align="center">7129</td><td align="center">86</td><td align="left">Lung adenocarcinoma survival</td></tr><tr><td align="left"><italic>Px-Bhattacharjee</italic></td><td align="center">2</td><td align="center">12600</td><td align="center">62</td><td align="left">Lung adenocarcinoma 4-year survival</td></tr><tr><td align="left"><italic>Px-Iizuka</italic></td><td align="center">2</td><td align="center">7070</td><td align="center">60</td><td align="left">Hepatocellular carcinoma 1-year recurrence-free survival</td></tr><tr><td align="left"><italic>Px-Pomeroy</italic></td><td align="center">2</td><td align="center">7129</td><td align="center">60</td><td align="left">Medulloblastoma survival</td></tr><tr><td align="left"><italic>Px-Rosenwald</italic></td><td align="center">2</td><td align="center">7399</td><td align="center">240</td><td align="left">Non-Hodgkin lymphoma survival</td></tr><tr><td align="left"><italic>Px-Veer</italic></td><td align="center">2</td><td align="center">24188</td><td align="center">97</td><td align="left">Breast cancer 5-year metastasis-free survival</td></tr><tr><td align="left"><italic>Px-Veer2</italic></td><td align="center">3</td><td align="center">24188</td><td align="center">115</td><td align="left">Breast cancer 5-year metastasis-free survival, metastasis within 5 years, germline BRCA1 mutation</td></tr><tr><td align="left"><italic>Px-Yeoh</italic></td><td align="center">2</td><td align="center">12240</td><td align="center">233</td><td align="left">Acute lymphocytic leukemia relapse-free survival</td></tr></tbody></table></table-wrap>
<inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M1" name="1471-2105-9-319-i1" overflow="scroll"><mml:semantics><mml:mrow><mml:msqrt><mml:mrow><mml:mi>n</mml:mi><mml:mi>u</mml:mi><mml:mi>m</mml:mi><mml:mi>b</mml:mi><mml:mi>e</mml:mi><mml:mi>r</mml:mi><mml:mo>⋅</mml:mo><mml:mi>o</mml:mi><mml:mi>f</mml:mi><mml:mo>⋅</mml:mo><mml:mi>g</mml:mi><mml:mi>e</mml:mi><mml:mi>n</mml:mi><mml:mi>e</mml:mi><mml:mi>s</mml:mi></mml:mrow></mml:msqrt></mml:mrow></mml:semantics></mml:math></inline-formula>
<supplementary-material content-type="local-data" id="S1"><caption><title>Additional file 1</title><p>Results for all gene selection methods, classifiers, and datasets.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S2"><caption><title>Additional file 2</title><p>Simulation experiment demonstrating sensitivity of random forests to input parameters.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S3"><caption><title>Additional file 3</title><p>Complete information about microarray datasets used in the study.</p></caption></supplementary-material>
<table-wrap-foot><p>The performance is estimated using area under ROC curve (AUC) for binary classification tasks and relative classifier information (RCI) for multicategory tasks. See subsection "Statistical comparison among classifiers" for the description of statistical test employed to compute reported p-values. P-values shown with boldface denote statistically significant differences between classification methods at the 0.05 α level.</p></table-wrap-foot> <table-wrap-foot><p>The performance is estimated using area under ROC curve (AUC) for binary classification tasks and relative classifier information (RCI) for multicategory tasks. See subsection "Statistical comparison among classifiers" for the description of statistical test employed to compute reported p-values. P-values shown with boldface denote statistically significant differences between classification methods at the 0.05 α level.</p></table-wrap-foot> <table-wrap-foot><p>Average number of genes selected over 10 cross-validation training sets.</p></table-wrap-foot> <table-wrap-foot><p>The reference paper for each dataset is provided in the Additional File ##SUPPL##2##3##.</p></table-wrap-foot>
<graphic xlink:href="1471-2105-9-319-1"/> <graphic xlink:href="1471-2105-9-319-2"/>
<media xlink:href="1471-2105-9-319-S1.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2105-9-319-S2.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2105-9-319-S3.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media>
[{"surname": ["Breiman"], "given-names": ["L"], "article-title": ["Random forests"], "source": ["Machine Learning"], "year": ["2001"], "volume": ["45"], "fpage": ["5"], "lpage": ["32"]}, {"surname": ["Lee", "Lee", "Park", "Song"], "given-names": ["JW", "JB", "M", "SH"], "article-title": ["An extensive comparison of recent classification tools applied to microarray data"], "source": ["Computational Statistics & Data Analysis"], "year": ["2005"], "volume": ["48"], "fpage": ["869"], "lpage": ["885"]}, {"surname": ["Rifkin", "Mukherjee", "Tamayo", "Ramaswamy", "Yeang", "Angelo", "Reich", "Poggio", "Lander", "Golub", "Mesirov"], "given-names": ["R", "S", "P", "S", "CH", "M", "M", "T", "ES", "TR", "JP"], "article-title": ["An analytical method for multi-class molecular cancer classification"], "source": ["SIAM Reviews"], "year": ["2003"], "volume": ["45"], "fpage": ["706"], "lpage": ["723"]}, {"surname": ["Sindhwani", "Bhattacharyya", "Rakshit"], "given-names": ["V", "P", "S"], "article-title": ["Information Theoretic Feature Crediting in Multiclass Support Vector Machines"], "source": ["Proceedings of the First SIAM International Conference on Data Mining"], "year": ["2001"]}, {"surname": ["Ling", "Huang", "Zhang"], "given-names": ["CX", "J", "H"], "article-title": ["AUC: a statistically consistent and more discriminating measure than accuracy"], "source": ["Proceedings of the Eighteenth International Joint Conference of Artificial Intelligence (IJCAI)"], "year": ["2003"]}, {"surname": ["Fawcett"], "given-names": ["T"], "article-title": ["ROC Graphs: Notes and Practical Considerations for Researchers"], "source": ["Technical Report, HPL-2003-4, HP Laboratories"], "year": ["2003"]}, {"surname": ["Efron", "Tibshirani"], "given-names": ["B", "R"], "article-title": ["Improvements on cross-validation: the .632+ bootstrap method"], "source": ["Journal of the American Statistical Association"], "year": ["1997"], "volume": ["92"], "fpage": ["548"], "lpage": ["560"]}, {"surname": ["Hastie", "Tibshirani", "Friedman"], "given-names": ["T", "R", "JH"], "source": ["The elements of statistical learning: data mining, inference, and prediction"], "series": ["Springer series in statistics"], "year": ["2001"], "publisher-name": ["New York, Springer"]}, {"surname": ["Hammer", "Gersmann"], "given-names": ["B", "K"], "article-title": ["A Note on the Universal Approximation Capability of Support Vector Machines"], "source": ["Neural Processing Letters"], "year": ["2003"], "volume": ["17"], "fpage": ["43"], "lpage": ["53"]}, {"surname": ["Dudoit", "Fridlyand", "Speed"], "given-names": ["S", "J", "TP"], "article-title": ["Comparison of discrimination methods for the classification of tumors using gene expression data"], "source": ["Journal of the American Statistical Association"], "year": ["2002"], "volume": ["97"], "fpage": ["77"], "lpage": ["88"]}, {"surname": ["Dietterich"], "given-names": ["TG"], "article-title": ["Ensemble methods in machine learning"], "source": ["Proceedings of the First International Workshop on Multiple Classifier Systems"], "year": ["2000"], "publisher-name": ["New York, NY, Springer-Verlag"], "fpage": ["1"], "lpage": ["15"]}, {"surname": ["Segal"], "given-names": ["MR"], "article-title": ["Machine Learning Benchmarks and Random Forest Regression"], "source": ["Technical Report, Center for Bioinformatics & Molecular Biostatistics, University of California, San Francisco"], "year": ["2004"]}, {"surname": ["Scheffer"], "given-names": ["T"], "source": ["Error estimation and model selection"], "year": ["1999"], "publisher-name": ["Ph.D.Thesis, Technischen Universit\u00e4t Berlin, School of Computer Science"]}, {"surname": ["Vapnik"], "given-names": ["VN"], "source": ["Statistical learning theory"], "series": ["Adaptive and learning systems for signal processing, communications, and control"], "year": ["1998"], "publisher-name": ["New York, Wiley"]}, {"surname": ["Fan", "Chen", "Lin"], "given-names": ["RE", "PH", "CJ"], "article-title": ["Working set selection using second order information for training support vector machines"], "source": ["Journal of Machine Learning Research"], "year": ["2005"], "volume": ["6"], "fpage": ["1918"]}, {"surname": ["Liaw", "Wiener"], "given-names": ["A", "M"], "article-title": ["Classification and regression by randomForest"], "source": ["R News"], "year": ["2002"], "volume": ["2"], "fpage": ["18"], "lpage": ["22"]}, {"surname": ["Breiman"], "given-names": ["L"], "article-title": ["Manual on setting up, using, and understanding Random Forests v4.0"], "year": ["2003"]}, {"surname": ["Guyon", "Weston", "Barnhill", "Vapnik"], "given-names": ["I", "J", "S", "V"], "article-title": ["Gene selection for cancer classification using support vector machines"], "source": ["Machine Learning"], "year": ["2002"], "volume": ["46"], "fpage": ["389"], "lpage": ["422"]}, {"surname": ["Chen", "Zeng", "van Alphen"], "given-names": ["X", "X", "D"], "article-title": ["Multi-class feature selection for texture classification"], "source": ["Pattern Recognition Letters"], "year": ["2006"], "volume": ["27"], "fpage": ["1685"], "lpage": ["1691"]}, {"surname": ["Menke", "Martinez"], "given-names": ["J", "TR"], "article-title": ["Using permutations instead of student's t distribution for p-values in paired-difference algorithm comparisons"], "source": ["Proceedings of 2004 IEEE International Joint Conference on Neural Networks"], "year": ["2004"], "volume": ["2"], "fpage": ["1331"], "lpage": ["1335"]}, {"surname": ["Good"], "given-names": ["PI"], "source": ["Permutation tests: a practical guide to resampling methods for testing hypotheses"], "series": ["Springer series in statistics"], "year": ["2000"], "edition": ["2nd"], "publisher-name": ["New York, Springer"]}]
BMC Bioinformatics. 2008 Jul 22; 9:319