<title>Background</title> <p>Various gene expression signatures have been extracted from breast cancer microarray data [##REF##11823860##1##,##REF##15721472##2##] to predict outcome. Subsequently, these signatures have been validated in larger cohorts [##REF##12490681##3##,##REF##17545524##4##], the datasets were re-analyzed to assess their robustness [##UREF##0##5##], or other data mining techniques were tested [##REF##15448693##6##,##REF##17940530##7##]. Depending on the technique used, different signatures are extracted from the same dataset. Often, these signatures have varying degrees of overlap in genes that compose the signature [##REF##17585334##8##]. This has led to the insight that there is no unique signature [##REF##15308542##9##].</p> <p>From a clinical point of view, accuracy of a signature is very important [##REF##18034879##10##]. To extract such an accurate gene expression signature, which is predictive of outcome, a variety of methodologies have been proposed [##REF##12011421##11##,##REF##11983868##12##]. Typically, the group of patients is dichotomized based on a five year survival threshold, resulting in a poor and a good outcome group. This assignment then serves as a class label, and the aim of this work is to predict this label. Most protocols apply a cross validation scheme, with which the generalization error and optimal number of genes are estimated. A final classifier is obtained by selecting the optimal number of genes and training the final classifier on all samples.</p> <p>Michiels <italic>et al. </italic>[##UREF##0##5##] introduced the concept of repeated random resampling to estimate the robustness (stability) of proposed signatures. The strategy relies on repeatedly analyzing a subset of patients in a dataset (a resampling), and then combining the results of the separate resamplings into a final list of genes. This strategy has been employed to investigate the stability of proposed signatures, and produces, by repeated resampling, a gene list which is employed as a type of 'gold standard'. This 'gold standard' has been shown to be different from signatures previously reported on the same data [##UREF##0##5##,##REF##16489042##13##,##REF##17894856##14##].</p> <p>However, the repeated random resampling approach has several drawbacks. First, resampling selects a subset of the available data, thereby reducing the number of samples and intensifying the small sample size problem. Second, the actual ranks within each resampling are not used (only presence in the top <italic>N </italic>genes is employed as a quality measure), which implies that the available information is not fully exploited. Lastly, the true rank of genes is unknown, making it hard to show that this strategy indeed provides a better ranking.</p> <p>We set up an experiment using artificial datasets to test the resampling strategy in a controlled environment. This allowed us to compare the resampling based ranking to the true ranking. In this comparison, we included two variants of the resampling strategy, and a non-resampling based strategy. On a higher level, datasets from different institutions consisting of samples from the same disease type, can be viewed as resamplings from the same underlying population. Concurrent analysis of these datasets aimed at, for example, producing a classifier, can be performed using a range of integration strategies. From a statistical point of view, a larger dataset implies more statistical power, advocating an early integration approach where (normalized) datasets are pooled prior to the analysis [##REF##15448693##6##]. A classifier derived from such a pooled dataset, should, in theory, be superior to those derived from a single dataset. Of course, in order to minimize inter-dataset variability, one of a variety of normalization techniques should be employed (e.g. Z-score normalization per feature per dataset) prior to pooling the datasets. Alternatively, a rank-based classification methodology has been put forward [##REF##16131522##15##,##REF##17663766##16##], which is independent of dataset normalization. Moreover, heterogeneity in the different datasets might be a hurdle [##UREF##1##17##], which might not be overcome with simple normalization approaches. In such cases, performing a meta-analysis may be advantageous, which implies combining statistics from each dataset. Lastly, 'late integration' (also termed 'combining decisions') involves training a classifier on the separate datasets and then merging the separate decisions to reach a final result. In fact, this approach possesses the least statistical power, since each of the individual classifiers have no benefit from the other datasets.</p> <p>A setting using artificial datasets, allows for a controlled analysis of the effects observed when pooling these artificial datasets. If synergy exists between the artificial datasets, we hypothesize that the classifier performance should improve when more artificial datasets are pooled. Therefore, we extended our analysis on artificial data to multiple artificial datasets. First of all, we start pooling pairs of artificial datasets, and thereafter increase the number of pooled datasets pooled until all six artificial datasets are pooled. We explored the effect of pooling artificial datasets on the classification performance (double loop cross validation), independent validation, and signature size.</p> <p>Currently, multiple breast cancer datasets are publicly available [##REF##11823860##1##,##REF##15721472##2##,##REF##16141321##18##,##REF##16936776##19##]. Teschendorff <italic>et al. </italic>[##REF##17076897##20##] created a consensus ER positive classifier from three breast cancer datasets employing a meta-analysis strategy.</p> <p>These datasets were measured on different platforms, thereby introducing difficulties with respect to probe matching, and the fact that different reference pools may have been used. If direct pooling of the datasets would be allowed, such a meta-analysis would not yield the optimal result. Moreover, true synergy of the meta-analysis on multiple datasets wasn't shown. Similarly, Xu <italic>et al. </italic>[##REF##18304324##21##] have created a consensus rank-based classifier from three different datasets, which was validated on a fourth independent dataset. To validate the type of effects observed when pooling artificial datasets, we applied the same analysis to a compendium of six breast cancer datasets (Desmedt <italic>et al. </italic>[##REF##17545524##4##], Minn <italic>et al. </italic>[##REF##16049480##22##], Miller <italic>et al. </italic>[##REF##16141321##18##], Pawitan <italic>et al. </italic>[##REF##16280042##23##], Loi <italic>et al. </italic>[##REF##17401012##24##], Chin <italic>et al. </italic>[##REF##17157792##25##]). This compendium of breast cancer datasets is particularly suited for such an analysis, since they were all measured on the same platform (Affymetrix HG U133A). In this setup, we inspected the effect of pooling breast cancer datasets on the classification performance (double loop cross validation), independent validation (a seventh dataset), signature size, and functional enrichment.</p> <p>Lastly, the analysis on such a compendium of six breast cancer datasets, might provide answers to an important unsolved question: why is there only limited overlap between existing signatures? [##REF##17585334##8##] As potential explanations, several explanations have been postulated, none of which have up to now been rigorously tested. We employ the results obtained when progressively pooling more breast cancer datasets, to provide insight into the correctness of the five explanations, and identify sample size as the most likely cause of limited signature overlap.</p>
<title>Methods</title> <title>Resampling procedure</title> <p>The resampling strategy has been proposed for the derivation of robust signatures [##UREF##0##5##,##REF##16489042##13##]. Briefly, the strategy works as follows. First, a subset of <italic>s</italic>% of the samples is selected, stratified relative to the outcome variable of interest. Second, for this subset all available genes <italic>G </italic>are ranked based on an appropriate selection statistic (e.g. the signal to noise ratio). Third, for each of <italic>R </italic>resamplings the top <italic>N </italic>genes are stored. The frequency of a gene being among the top <italic>N </italic>genes determines its final rank.</p> <p>All genes with a non-zero frequency of appearing in the top <italic>N </italic>genes at least once can be considered part of the resulting robust signature [##REF##16489042##13##]. However, when assessing a large number of resamplings, one would expect a fair number of genes to achieve a non-zero frequency by chance. To remove these genes, we propose to apply a threshold. Under the null hypothesis that there is no information in the dataset, we assume that each of the resamplings is a Bernouilli trial (with ). Under the null hypothesis, how often a gene appears in the top <italic>N </italic>across the <italic>R </italic>resamplings follows a Binomial distribution, with an expected value (<italic>E</italic>) of . Consequently, when assessing the frequency of occurrence in the top <italic>N</italic>, we divide by <italic>R </italic>again, leading to the following threshold <italic>t</italic>:</p> <p></p> <p>In our analysis, we used the resampling strategy in conjunction with the SNR (signal to noise ratio) as a ranking criterion. Assuming a dataset <italic>A</italic>, the absolute SNR is defined as:</p> <p></p> <p>where <italic>SNR</italic>(<italic>i</italic>) represents the SNR of gene <italic>i</italic>, <italic>μ</italic>(<italic>A</italic><sub><italic>i</italic>1</sub>) represents the mean of gene <italic>i </italic>for all samples in the class 1, and <italic>σ</italic>(<italic>A</italic><sub><italic>i</italic>1</sub>) represents the standard deviation of gene <italic>i </italic>for all samples in the class 1.</p> <title>Double loop cross validation</title> <p>Wessels <italic>et al. </italic>[##REF##15817694##26##] described a generally applicable framework for building diagnostic classifiers from high throughput data. We adopted this methodology combined with forward filtering as feature selector, the signal to noise ratio as criterion to evaluate the individual genes, and a nearest mean classifier (cosine correlation as distance measure). The training and validation procedure was performed employing 20 repeats of 5 folds cross validation in the outer (validation) loop, and 10 fold cross validation in the inner loop. Learning curves were constructed for up to 200 genes. At all points data splits were stratified with respect to the class prior probabilities.</p> <p>The double loop cross validation method can be described in a few steps:</p> <p>1. For each repeat, the data is split (stratified) into five parts (different splits for each repeat).</p> <p>2. For each fold, four parts are used for the inner loop (training set), the fifth is used in the outer loop for validation (validation set).</p> <p>3. On the training set data, a 10-fold cross validation is performed to estimate the optimal number of genes <italic>n</italic>.</p> <p>4. Next, a classifier is trained on the complete training set, using the top <italic>n </italic>genes.</p> <p>5. Finally, the performance of that classifier is assessed on the validation set.</p> <p>6. After all repeats are completed, a final classifier is created by ranking the genes using all samples. The average number of <italic>n </italic>that were obtained are used to train the classifier. This classifier is then applied on an external validation set.</p> <p>Typically, datasets are unbalanced with respect to the class priors. Moreover, the class priors will be different for different datasets. Hence, directly comparing overall error rates (fraction of wrong assignments), is not an appropriate comparative measure. Therefore, classification errors were calculated by using the average false positive false negative ratio (which takes the class priors into account), which is defined as:</p> <p></p> <p>where <italic>e</italic><sub><italic>FPFN </italic></sub>represents the average false positive false negative ratio error, <italic>TP </italic>the number of true positives, and <italic>TN </italic>the number of true negatives. This ratio is equivalent to 1-.5(Sensitivity + Specificity). In each iteration in the inner loop, <italic>n </italic>is defined as the number of genes at which the <italic>e</italic><sub><italic>FPFN </italic></sub>is minimal.</p> <p>When we refer to the DLCV error, this is the validation error assessed in the outer loop of the protocol. In addition to the linear nearest mean classifier, we also performed the experiments using more complex, non-linear classifiers. To this end, we used a k-nearest neighbor classifier (K-NN) as well as a Support Vector Machine with a Radial Basis Function as kernel (SVM with RBF kernel). The K-NN classifier was run using <italic>K </italic>= 3 and the cosine correlation as distance measure. For the SVM we used the following parameters: <italic>C </italic>= 5 and <italic>λ </italic>= 2 (RBF kernel width). These two classifiers were used in conjunction with the same double loop cross validation protocol.</p> <title>Artificial dataset</title> <p>An artificial dataset was used to be able to compare the ranking obtained by different strategies to a known ground truth. To this end, we generate a dataset which is comparable to microarray datasets in terms of dataset size. We constructed an artificial dataset with <italic>M </italic>samples and <italic>G </italic>genes. The data for each feature was sampled from the following two class conditional density functions:</p> <p></p> <p></p> <p>where</p> <p></p> <p>The data that is generated using this model is more ideal than real-life microarray data, since it does not include any covariance terms. Artificial datasets were generated with the following setting of the parameters:</p> <p></p> <p></p> <p></p> <p></p> <p></p> <p></p> <title>Breast cancer datasets</title> <p>The data from Veer <italic>et al. </italic>[##REF##11823860##1##] was downloaded from the repository mentioned in the paper. Preprocessing was done as follows, the 24481 probes were pruned to 4919 by using only probes with at least a twofold change and <italic>p </italic>&lt; 0.01 for three or more arrays [##REF##11823860##1##].</p> <p>We used a collection of six available datasets containing microarray data of breast cancer samples. These datasets were all measured on Human Genome HG U133A Affymetrix arrays and normalized using the same protocol. The datasets were downloaded from NCBI's Gene Expression Omnibus (GEO, <ext-link ext-link-type="uri" xlink:href="http://www.ncbi.nlm.nih.gov/geo/"/>) with the following identifiers; GSE6532 [##REF##17401012##24##], GSE3494 [##REF##16141321##18##], GSE1456 [##REF##16280042##23##], GSE7390 [##REF##17545524##4##] and GSE5327 [##REF##16049480##22##]. The Chin <italic>et al. </italic>[##REF##17157792##25##] data set was downloaded from ArrayExpress (<ext-link ext-link-type="uri" xlink:href="http://www.ebi.ac.uk/"/>, identifier E-TABM-158).</p> <p>To ensure comparability between the different datasets, they were all subjected to the same pre-processing procedure. Microarray quality-control assessment was carried out using the R AffyPLM package available from the Bioconductor web site (<ext-link ext-link-type="uri" xlink:href="http://www.bioconductor.org"/>, [##UREF##2##50##]). We applied the Relative Log Expression (RLE) test and the Normalized Unscaled Standard Errors (NUSE) test. Chip pseudo-images were produced to assess artifacts on arrays that didn't pass the preceding quality control tests. Selected arrays were normalized according to a three step procedure using the RMA expression measure algorithm (<ext-link ext-link-type="uri" xlink:href="http://www.bioconductor.org"/>, [##REF##12582260##51##]): RMA background correction convolution, median centering of each gene across arrays separately for each data set and quantile normalization of all arrays.</p> <p>For each sample, the ER status was determined by the expression profile [##REF##17329190##52##]. This ensures a consistent assignment of the ER status across all samples, and rules out any inconsistencies in the assignments by pathologists. In addition, for part of the samples the IHC ER status was unavailable (see Additional file ##SUPPL##13##14##). All samples were ranked based on the expression level of the ER probe (probe 205225_at), after which we constructed an ROC curve using the IHC (Immuno-histo chemistry) assignment as true label. We then selected the threshold which resulted in the smallest error with regard to the true label (ERneg when 205225_at ≤ -1.84, ERpos when 205225_at &gt; -1.84).</p> <p>For the datasets from Desmedt, Miller, Pawitan, and Loi, there is an overlap in terms of samples. Part of these samples have been re-used, either with the same data (overlap Loi/Miller), or after a new hybridization (overlap Desmedt/Miller). We excluded all overlapping samples from the Desmedt and Loi datasets. This way, we ensured independence of the datasets, and avoid any bias.</p> <p>We used the dataset from Vijver <italic>et al. </italic>[##REF##12490681##3##] as an independent validation. This dataset was measured on the Agilent platform.</p> <p>To validate the classifier trained on the six Affymetrix datasets on the Agilent dataset, we had to map the two probesets. We did this by mapping the probes from both platforms to the set of Entrez ids (11751) present on both. For the HG U133A array, only probes with an '_at', '_s_at', and '_x_at' extension were used (and prioritized in this order). When multiple probes match the same Entrez id (and had the same extension), the one with the highest variance was used.</p> <p>Poor and good subgroups were defined as follows: Poor refers to samples with an event within five years, whereas Good refers to samples with event free survival and at least five years follow up.</p> <p>Table ##TAB##0##1## provides an overview of the datasets that were used.</p> <title>Functional enrichment analysis</title> <p>For all signatures, we evaluated whether specific gene sets (i.e. functional groups), are overrepresented. We gathered a collection of 5480 gene sets from four databases: Gene Ontology (<ext-link ext-link-type="uri" xlink:href="http://www.ncbi.nlm.nih.gov/"/>, Function and Process trees, 4745 gene sets), KEGG (<ext-link ext-link-type="uri" xlink:href="http://www.genome.ad.jp"/>,187genesets), Reactome (<ext-link ext-link-type="uri" xlink:href="http://www.reactome.org"/>, 26 gene sets) and the MSDB (<ext-link ext-link-type="uri" xlink:href="http://www.broad.mit.edu/gsea/"/>, C2, 522 gene sets). For the enrichment analysis we only used gene sets with at least five annotated Entrez IDs that are also present on the array, resulting in 1860 gene sets. We used the hypergeometric test to test the significance of the overlap between each signature and gene set. Multiple testing correction was taken into account by applying a Bonferroni correction (per signature).</p>
<title>Results and discussion</title> <title>Resampling on a single breast cancer dataset</title> <p>We applied the repeated random resampling strategy (see Methods section) on the Veer <italic>et al. </italic>[##REF##11823860##1##] dataset, see Figure ##FIG##0##1##. When comparing the frequency-based ranking to the set of genes in the published 70-gene and 231-gene signatures, there appear to be additional genes with high frequencies of occurrence that were not part of the published signatures. Both Michiels <italic>et al. </italic>[##UREF##0##5##], and Roepman <italic>et al. </italic>[##REF##16489042##13##] point out that the genes with high frequencies of occurrence, which are not part of the original signature, are of high interest. They hypothesize that these genes were probably not picked up in the signatures due to sampling specific biases in the dataset. This suggests a beneficial effect due to the resampling which does pick them up. However, since the true ranking of genes underlying breast cancer outcome is unknown, a plausible alternative hypothesis could be that those genes are false positives.</p> <p>Roepman <italic>et al. </italic>[##REF##16489042##13##] considers all genes with a non-zero frequency of occurrence to represent a robust signature. This seems rather optimistic, since genes will be expected to have a positive frequency by chance. Therefore, we propose to set a threshold to prune the set of genes to only those with a frequency higher than expected by chance (see Methods section). This threshold is indicated by the vertical red line in Figure ##FIG##0##1##. The set of genes retained after this pruning step still contains all genes from the original signatures, i.e. all light-blue bars (the original signatures) reside on the left of the threshold. Of course, the ranking itself remains unchanged.</p> <p>In the current experimental setting, it is impossible to conclude which hypothesis correctly describes the appearance of genes with a high (above the threshold) frequency of occurrence, that were not in the original signature.</p> <title>Resampling on a single artificial dataset</title> <p>We set up an experiment on artificial data to be able to compare the ranking after resampling to a known ground truth (see Methods section). The artificial dataset mimics a microarray dataset, and consists of 5000 genes, of which either 500, 2500 or 5000 genes were informative. The experiments were done using artificial datasets consisting of 100 to 1000 samples, with equal priors for the two classes. The data was sampled from a class-conditional gaussian distribution, without any covariance structure. Since genes are assumed to be independent, these datasets are more ideal than real microarray datasets. Nevertheless, this controlled, artificial setting can be a useful tool to gain insight into the relationship between sample size and gene selection.</p> <p>On these artificial datasets we compare the ranking resulting from the resampling strategy with the ranking based on all samples. Additionally, we also consider two variants of the resampling strategy. Instead of only storing the top <italic>N </italic>genes in each of the resamplings, we sum the statistics (sum of ranks or sum of signal-to-noise ratios (SNRs)) of the genes across all the resamplings, and base the final ranking on this cumulative statistic. Summing the statistics amounts to a 'meta-analysis', and the Top <italic>N </italic>method is a 'combining decisions' approach. We then compare the ranking obtained by each method to the ground truth (Spearman rank correlation). Since a particular initialization of the artificial data might influence the results, we repeated the entire experiment 100 times, and present statistics across all repeats (average and standard deviation). Figure ##FIG##1##2## shows the Spearman rank correlation between the known ranks of the artificial data and those obtained by the Top <italic>N </italic>resampling approaches, the two resampling variants, and the ranking over all samples. For <italic>N </italic>we considered values of 50, 100 and 200, similar to Michiels <italic>et al. </italic>[##UREF##0##5##], and Roepman <italic>et al. </italic>[##REF##16489042##13##].</p> <p>Figure ##FIG##1##2## indicates that the ranking based on all samples outperforms the ranking obtained from resampling and then determining the frequency of being in the top <italic>N</italic>. Furthermore, it is evident that combining the statistics (sum of ranks or sum of SNRs) of each resampling performs similar to the ranking over all samples. In fact, it is quite straight-forward to show analytically that, for a given gene, the summation of SNRs across resamplings is equal to the SNR over all samples (see Additional file ##SUPPL##0##1##). This is true when enough resamplings are performed, and they are uniformly sampled from the complete dataset.</p> <p>For smaller artificial datasets the obtained Spearman rank correlations are lower compared to larger artificial datasets. The reduced sample size is the most plausible explanation for this effect. For larger artificial datasets, the gap in performance between the top <italic>N </italic>resampling approaches and the sum of ranks and sum of SNRs widens, indicating that the latter two methods profit more from an increase in artificial dataset size.</p> <p>Taken together, these results clearly show that the random resampling strategy produces a suboptimal ranking of the prognostic genes, and that it is preferable to employ a ranking based on all samples in the dataset. In addition, these differences become more pronounced as the sample size increases. Given that these effects are already evident on an artificial dataset, we hypothesize that the sample size effects will be even stronger on data with a complex covariance structure. We therefore conclude that it is unwise to use the repeated random resampling approach on real data for ranking features.</p> <title>Pooling artificial datasets</title> <p>The experiment on a single artificial dataset is easily extendible to a scenario where multiple artificial datasets are available. Such a setup would emulate the availability of several microarray datasets from the same biological context. We hypothesize that a significant association exists between the classification error and the number of datasets pooled.</p> <p>We set up an experiment using six artificial datasets, each with 200 samples and 5000 genes, of which 500 are informative. Each of these six artificial datasets were generated using the same model, and no additional noise or heterogeneity was introduced. This represents the ideal case, where each of the six datasets are truly relevant for the same underlying problem. We applied the double loop cross validation (DLCV, nearest mean classifier) protocol on each artificial dataset separately (see Methods section), and each possible pooled subset of artificial datasets. First of all, we inspected the behavior of pooling pairs of artificial datasets relative to the performance on the individual artificial datasets. A DLCV error which is lower for the pooled pair compared to the individual errors implies a synergetic effect on the classification performance. When the pooled DLCV error is between the two individual errors, and lower/higher than the weighted mean (number of samples chosen as weights) we label the effect as marginal synergetic/marginal anti-synergetic. Conversely, if the DLCV error is strictly higher than the individual errors it is labeled as anti-synergetic. Figure ##FIG##2##3## shows a network indicating the type of synergy that was observed for the six artificial datasets, labeled Art1 to Art6.</p> <p>Figure ##FIG##2##3## shows that there is a synergetic effect on the DLCV error for 14 of the 15 pooled pairs of artificial datasets. A marginal anti-synergetic effect is only observed in one of the 15 cases. This suggests a large gain in statistical power when pooling the artificial datasets.</p> <p>Next, we evaluated the DLCV error for every potential pooling of one to six artificial datasets. In addition, we evaluated the performance of each of the classifiers derived from the pooled data on an independent large validation set of 2000 artificial samples, see Figure ##FIG##3##4A## and ##FIG##3##4B##. Figure ##FIG##3##4A## and ##FIG##3##4B## show that a high degree of synergy is obtained by pooling artificial datasets. Both the DLCV error as well as the error on the validation set show a highly significant correlation with the number of artificial datasets that is pooled, Pearson correlation of -0.91 (<italic>p </italic>= 1.1<italic>e </italic>- 24) and -0.95 (<italic>p </italic>= 1.4<italic>e </italic>- 31), respectively.</p> <p>To test whether the observed trends are classifier specific, we repeated the experiment using non-linear classifiers (as opposed to the linear nearest mean classifier used above). Thus, we repeated the experiment using a K-Nearest Neighbor classifier (K-NN) as well as a Support Vector Machine with a Radial Basis Function as kernel (SVM with RBF kernel). Using these classifiers, we observed the same trends for the DLCV and Validation error. That is, in all cases the performance gets better after pooling the artificial datasets (trends significant, <italic>p </italic>&lt; 0.05). However, the errors obtained are higher than those from the NMC. The plots are shown in Additional files ##SUPPL##1##2## and ##SUPPL##2##3##.</p> <p>In addition to the classification performance, we looked at the optimal number of genes that is selected by the classifiers, see Figure ##FIG##3##4C##. It turns out that there is a highly significant positive trend (Pearson correlation 0.85, <italic>p </italic>= 1.8<italic>e </italic>- 18) towards selecting more genes when artificial datasets are pooled.</p> <p>In summary, pooling artificial datasets shows that larger training sets result in larger gene sets being selected and, most importantly, in better classification performance. Since this effect is already clearly present in the simple model, we believe that this effect will be even stronger in more complex models which include, for example, a covariance structure between genes. This stems from the fact that these more complex models will require more data (relative to the simple models) to accurately estimate the required complex classifiers.</p> <title>Pooling breast cancer datasets</title> <p>We used a compendium of six Affymetrix breast cancer datasets to evaluate the effect of pooling datasets on the performance of classifiers (see Table ##TAB##0##1## and methods section). Our aim is to classify patients into the poor/good outcome groups as well as possible. To this end, we followed the same strategy as on the artificial datasets. First, we inspected the synergy effect on the DLCV error achieved on pairs of datasets relative to the DLCV errors on the individual datasets (DLCV, nearest mean classifier). Figure ##FIG##4##5## shows a network indicating the synergy among the six real datasets.</p> <p>Figure ##FIG##4##5## shows that for 11 of the 15 pairwise pooled datasets a synergetic effect was observed (green/blue link). For 4 of the 15 pairwise pooled datasets there was a marginal anti-synergetic effect (yellow link), and for none of the pairs the effect was anti-synergetic (orange link).</p> <p>As a next step we applied the DLCV framework on each potential pooling of the datasets, and assessed the DLCV error. In addition, we evaluated the performance of each classifier on a seventh independent dataset (Vijver <italic>et al. </italic>[##REF##12490681##3##]). Figure ##FIG##5##6A## and ##FIG##5##6B## show the DLCV error and validation error as a function of the number of datasets pooled.</p> <p>There is a clear trend indicating that the DLCV error decreases when more datasets are pooled (see Figure ##FIG##5##6A##). The observed trend has a Pearson correlation of -0.26, which is significant: <italic>p </italic>= 4.0<italic>e </italic>- 2. Of course, since the different datasets have different numbers of samples, an alternative would be to evaluate the correlation of the error and the actual number of samples in the pooled datasets. Doing this unveils a similar significant correlation (Additional file ##SUPPL##3##4##).</p> <p>The results on the seventh independent validation set (Vijver <italic>et al. </italic>[##REF##12490681##3##] data) confirm the synergetic effect obtained by pooling datasets (see Figure ##FIG##5##6B##). The trend is even more pronounced at a Pearson correlation of -0.35, with <italic>p </italic>= 5.0<italic>e </italic>- 3. In addition, in a multivariate Cox proportional hazards model the signature is an independent predictor of outcome in the presence of the standard clinico-pathological variables (data not shown). Similarly, plotting the results in a Kaplan-Meier plot indicates a highly significant separation (<italic>p </italic>= 9.5<italic>e </italic>- 7, logrank test, data not shown).</p> <p>The nearest mean classifier that was used in the above experiment is a relatively simple linear classifier, which is known to perform well on this type of data [##REF##15817694##26##]. To test whether the observed trend is classifier specific, we repeated the experiment using non-linear classifiers. Thus, we repeated the experiment using a K-Nearest Neighbor classifier (K-NN) as well as a Support Vector Machine with a Radial Basis Function as kernel (SVM with RBF kernel). The results that were obtained for the DLCV error and the validation error on the Vijver dataset are shown in Additional files ##SUPPL##4##5## and ##SUPPL##5##6##, for the K-NN and SVM classifiers, respectively.</p> <p>Overall, the average eFPFN across all 63 different pooling combinations when using the K-NN classifier (DLCV = 0.356) is slightly worse compared to the nearest mean classifier (DLCV = 0.348). In addition, the error obtained using the SVM (DLCV = 0.391) is higher than the K-NN. Similarly, the average error on the independent validation set (Vijver) is also higher: (0.329 for the NMC, 0.365 for the K-NN and 0.405 for the SVM). This corresponds to the previous observation that simpler classifier often perform better on this type of data. At the same time, the trend that classifiers that are derived from pooled datasets are more accurate is also observed.</p> <p>In conclusion, not only simple linear classifiers, but also more complex non-linear classifiers show a clear benefit by training them on pooled datasets. Moreover, these results confirm that the linear nearest mean classifier is the best candidate for this type of classification problem.</p> <p>Previously reported classifiers have been using varying numbers of genes. Signatures that have been derived using a cross validation setup that is similar to the one used here, use 52 [##REF##17076897##20##] to around 70 genes [##REF##15721472##2##,##REF##12490681##3##]. We have investigated the number of genes selected by the DLCV method, for each of our pooled datasets, see Figure ##FIG##5##6C##. The classifiers trained on a single dataset use around 50 genes, whereas pooled datasets select progressively more genes. Especially the classifiers from five or six datasets select more than 100 genes. These results indicate a positive trend in the number of datasets pooled and the number of genes selected in the signatures (Pearson correlation of 0.38 with <italic>p </italic>= 1.9<italic>e </italic>- 3).</p> <p>In summary, the effects observed when pooling artificial datasets are also observed when pooling breast cancer datasets. This implies that datasets should be analyzed jointly rather than in isolation.</p> <title>Enrichment of the signatures</title> <p>Functional enrichment of prognostic signatures has unveiled a multitude of pathways that are associated with tumor progression. We set out to inspect the effect of pooling datasets on the enrichment. We used a collection of 1860 genes sets consisting of at least five annotated genes (see Methods section). For each pooled combination, we used the hypergeometric test to assess the significance of the overlap, followed by Bonferroni correction (see Methods section).</p> <p>The most highly enriched categories were proliferation associated, which are well-known to be associated with tumorigenesis. Figure ##FIG##6##7## shows the enrichment for three of the gene sets. Figure ##FIG##7##8## depicts a complete heatmap of enrichments.</p> <p>As seen in Figure ##FIG##6##7A##, the mitosis GO category (GO:0007067: mitosis) shows a significant enrichment for practically every pooled combination of datasets. This is an example of a very strong signal, which gets even stronger when pooling datasets. Another proliferation related category (Figure ##FIG##6##7B##), the cell cycle pathway (KEGG – hsa04110 – Cell cycle) becomes significantly enriched when roughly three or more datasets are combined. Thus it's association is weaker than the mitosis gene set, but gets detected when pooling a few datasets.</p> <p>Lastly, a gene set associated to microtubule activity (GO:0003777: microtubule motor activity) is non-significant for the range of pooling one to three datasets (Figure ##FIG##6##7C##). When combining four or five it is sometimes picked up, and is definitely picked up when pooling six datasets. This is a clear example of a relatively weak signal, which benefits significantly from pooling datasets.</p> <p>Conversely, we also observe that many gene sets which are significant in some of the single/pairs of datasets are no longer significant when pooling (Figure ##FIG##7##8##). For instance, glycolysis related pathways are only picked up in the signature extracted from the Pawitan <italic>et al. </italic>[##REF##16280042##23##] dataset. This is evidence that those categories are specific to this dataset, but represent false positives within the context of the global breast cancer population.</p> <p>Overall, the enrichment analysis provides further proof that pooling datasets significantly increases the statistical power to detect weak signals and filter out false positive associations.</p> <title>Heterogeneity amongst the breast cancer datasets</title> <p>Pooling datasets with a high degree of heterogeneity might have a detrimental effect on subsequent analysis. A potential way to limit the heterogeneity would be to restrict the analysis to known clinical subgroups. To test this, we split each dataset based on ER status, and repeated the analysis on each of these subgroups. Of course, these splits reduce the number of samples and intensify small sample size related problems. For this reason it becomes impractical to exhaustively explore all possible stratifications of the dataset. We chose the ER status since it is known to have a profound effect on disease progression and gene expression.</p> <p>On the subgroup of ER positive samples, we observed the same trends as on the complete datasets. We observed a synergetic effect for 10 of the 15 of the pooled pairs of datasets (see Additional file ##SUPPL##6##7##). Moreover, the trends observed on all samples are also significant on the ER positive subset (see Additional file ##SUPPL##7##8##). Enrichment analysis of these signatures revealed the same set of categories that were found on the complete set of samples (see Additional file ##SUPPL##8##9##).</p> <p>Similarly, we also repeated all analyses on the ER negative subgroup of samples. We observe the same trends as on all samples or the ER positive group (see Additional file ##SUPPL##9##10## and ##SUPPL##10##11##). However, the classification accuracy on this subgroup is much lower. A potential explanation for this observation is that this group of samples is inherently more difficult to classify, or that the small sample size related problems play a big role (combining all six datasets results in 191 samples).</p> <p>Enrichment analysis of the signatures from the ER negative group, leads to a very limited set of enriched gene sets (see Additional file ##SUPPL##11##12##). The most highly enriched category is 'GO:0006955: immune response'. This confirms previous reports on the association of this category with outcome in the ER negative group [##REF##17683518##27##,##REF##17910759##28##]. However, in the largest dataset (all ER negative samples), not a single category is enriched. In both of the ER groups, we observed the same synergetic trends when pooling datasets on cross validation, independent validation, and the number of genes selected. Therefore, heterogeneity among the datasets due to ER status is only limited.</p> <title>Overlap amongst signatures</title> <p>It has been previously observed that existing signatures have limited overlap in terms of genes. Several potential explanations have been proposed for these differences:</p> <p>1. Heterogeneity in expression due to different platform technologies and references;</p> <p>2. Differences in supervised protocols with which signatures are extracted;</p> <p>3. Although the genes are not exactly the same, they represent the same set of pathways;</p> <p>4. Differences in clinical composition between datasets (i.e. sample heterogeneity) and</p> <p>5. Small sample size problems that cause inaccurate signatures.</p> <p>Here we will critically evaluate whether we observe support for each of these explanations in the context of the six breast cancer datasets we studied.</p> <title>Explanations 1 and 2</title> <p>In our analysis, we have restricted ourselves to the same platform for training our classifiers, and use the same supervised protocol on all datasets. This allowed us to check whether the first two explanations apply. We evaluated the relative overlap of every combination of two signatures that were extracted from a single dataset, see Figure ##FIG##8##9A##. For these 15 comparisons, the majority of genes are exclusively part of one signature (mean overlap 1.8%). This clearly indicates that, within the context of the datasets studied here, Explanations 1 and 2 are not the likely causes of the limited overlap between the signatures from these six datasets.</p> <title>Explanation 3</title> <p>Using our functional enrichment analysis we checked Explanation 3. The enrichment patterns of the signatures from single datasets are largely disparate (Figure ##FIG##6##7##). Two of the six signatures are not enriched for any functional category. There is only one gene set (GO:0007049: cell cycle), which is enriched in three signatures, namely those derived from Desmedt <italic>et al. </italic>[##REF##17545524##4##], Miller <italic>et al. </italic>[##REF##16141321##18##], and Loi <italic>et al. </italic>[##REF##17401012##24##]. When considering the set of enriched gene sets that overlap between every pair of signatures (Figure ##FIG##8##9D##), we observed a mean overlap of less than 1%. Taken together, this indicates that there is very little support for the third explanation in these six datasets.</p> <title>Explanation 4</title> <p>Breast cancer is well known to be a complex disease which manifests itself in a range of subtypes. As a result, differences in the clinical composition of different datasets might explain the different signatures. However, when creating a histogram of signature overlap within the ER positive/negative subgroups we also found that the majority of genes are only part of one signature. Thus, heterogeneity with respect to ER status does not explain the limited degree of overlap among the signatures from these six datasets. Of course, it could still be that differences in the composition with respect to another (unknown) clinical parameter partly explain the lack of overlap.</p> <title>Explanation 5</title> <p>Signatures that are derived from pooled datasets have been derived using more samples, thereby easing small sample size related difficulties. Therefore, we inspected the relative overlap between signatures derived from one, two and three datasets. To avoid any bias, we only consider pairs of pooled datasets that do not overlap in terms of samples. The histogram and p-values shown in Figure ##FIG##8##9A–C## represent the mean over all these combinations. The overlap becomes progressively larger for signatures that are extracted from larger datasets (single dataset: 1.8%, two pooled datasets: 8.1%, three pooled datasets: 13.1%). At the same time this overlap becomes more significant (single dataset: -<italic>log</italic>10(<italic>p</italic>) = 2.1, two pooled datasets: -<italic>log</italic>10(<italic>p</italic>) = 13.6, three pooled datasets: -<italic>log</italic>10(<italic>p</italic>) = 23.2). The same trends were observed on the artificial data, and the ER positive subgroup (results not shown). Next, we repeated this analysis by looking for overlap in terms of enriched gene sets (Figure ##FIG##8##9D–F##). Similarly, signatures derived from the pooled datasets show a higher degree of enriched gene sets that overlap. The distinguishing property of these signatures is the fact that they were derived from a larger dataset, i.e. more samples. Thus, we conclude that Explanation 5 is the most relevant explanation for the low degree of overlap in genes from signatures derived from these six datasets.</p> <title>Consensus signature across six datasets</title> <p>The signature from the six pooled datasets will be substantially different from those derived from each individual dataset. To gain insight into the difference, we created a chart indicating the rank position of all 127 genes selected in the classifier trained on all six datasets, along with rank positions from these genes in the rankings obtained from the individual datasets (Figure ##FIG##9##10##). Additional file ##SUPPL##12##13## contains detailed information on the 127 genes, and the centroids of the classifier. Most of the 127 signature genes, had a much lower rank (larger rank number) in the individual signatures, implying that they're not part of those signatures. The difference in ranking of the 127 genes is especially large compared to the signature derived from the Minn <italic>et al. </italic>[##REF##16049480##22##] and Chin <italic>et al. </italic>[##REF##17157792##25##] data. This implies that those rankings are dominated by genes which are not generally relevant for breast cancer.</p> <p>We compared our signature to signatures derived from different breast cancer datasets. We compared to the 70-gene signature from Wang <italic>et al. </italic>[##REF##15721472##2##], 76-gene signature from Vijver <italic>et al. </italic>[##REF##12490681##3##], 70-gene signature from Naderi <italic>et al. </italic>[##REF##16936776##19##], and the 52-gene signature from Teschendorff <italic>et al. </italic>[##REF##17076897##20##]. The latter of which was derived from the first three datasets. None of these three datasets were employed to derive the 127 gene signature. The overlap between the 127 genes and the Teschendorff signature is more significant (<italic>p </italic>= 2.3<italic>e </italic>- 18) than with Naderi (<italic>p </italic>= 1.2<italic>e </italic>- 11), Vijver (<italic>p </italic>= 7.4<italic>e </italic>- 7) or Wang (<italic>p </italic>= 4.7<italic>e </italic>- 1). This provides additional evidence that the limited signature overlap is most likely caused by small sample size difficulties.</p> <p>The set of genes in the signature from all six datasets contains many well known and described genes. For instance, it confirms the significance of HMMR (Rank 127, Figure ##FIG##9##10##) which was only recently associated with breast cancer [##REF##17922014##29##]. This gene was not picked up in any of the individual signatures. In the same study, Pujana <italic>et al. </italic>[##REF##17922014##29##] derived a set of 164 genes that are associated with BRCA1, BRCA2, ATM, and CHEK2. This set of 164 genes is significantly overrepresented in the 127 signature genes (<italic>p </italic>= 2.4<italic>e </italic>- 9).</p> <title>Biological relevance of the consensus signature</title> <p>Cell cycle and Mitosis pathways are the common background of most prognostic gene expression classifiers. Not surprisingly, many of the 127 genes have been previously described as markers of proliferation and poor prognosis in different types of cancer. These include: BUB1B, BUB1, MAD2L1, C16orf61 (DC13), CCT5, EIF4EBP1, MELK, MYBL2, CCNA2, CCNB1 and CCNB2. More specifically, EIF4EBP1 (4E-binding protein 1, 4E-BP1) is a EIF4E-binding protein that plays a critical role in the control of protein synthesis, survival, and cell growth. Rojo <italic>et al. </italic>[##REF##17200342##30##] showed that 4EBP1 is activated in a high percentage of breast tumors, and is associated with higher malignant grade, tumor size, and local recurrence regardless of HER2/neu status (Armengol <italic>et al. </italic>[##REF##17699757##31##]). Lin <italic>et al. </italic>[##REF##17280616##32##] showed that MELK (maternal embryonic leucine zipper kinase), is overexpressed in both breast cancer specimens and cancer cell lines, and that its kinase activity possibly plays a significant role in breast cancer cell proliferation. They show that down-regulation of MELK by treatment with siRNA significantly suppressed the cell growth of breast cancer, indicating its crucial role in the proliferation of breast cancer.</p> <p>C16orf61 (DC13) and CCT5 are examples of genes associated with therapy resistance. More specifically, C16orf61 (DC13) was previously found to be involved in multidrug resistance in breast cancer [##REF##17659439##33##], and mRNA levels of CCT5 were significantly associated with resistance to docetaxel [##REF##16821082##34##].</p> <p>Cellular movement and cytokinesis pathways are significantly over-represented in our prognostic signature mainly through genes (KIF23, KIFC1, KIF14, KIF20A, KIF2C, KIF13B, KIF4A, PRC1, TUBB3, TPX2, CENP-A) linked to the microtubule motor activity (GO:0003777) and the microtubule-based movement (GO:0007018) GO categories. It is of major interest that these GO categories are non-significant for the range of pooling one to three datasets and are only picked up when pooling more datasets.</p> <p>The genes that are associated with the microtubule categories are all part of the kinesin family. This family is composed of more than 40 genes involved in spindle assembly and function, chromosome segregation, mitotic checkpoint control and cytokinesis. They are motor proteins that generate movement along microtubules. KIF14 is a prognostic factor in breast and lung cancer (Corson <italic>et al. </italic>[##REF##16570270##35##,##REF##17545527##36##]). KIF20A is over-expressed in pancreatic ductal adenocarcinoma (PDAC) and siRNA knockdown of KIF20A expression in PDAC cell lines attenuated growth of those cells (Taniuchi <italic>et al. </italic>[##REF##15665285##37##]). KIF2C is also known to be over-expressed in breast cancer cells. Treatment of breast cancer cells (T47D and HBC5) by siRNA against KIF2C suppressed KIF2C expression and inhibited the growth of the breast cancer cell lines (Shimo <italic>et al. </italic>[##REF##17944972##38##]). Immunoprecipitation assay showed an interaction between PRC1 and KIF2C in breast cancer cells (Shimo <italic>et al. </italic>[##REF##17944972##38##]). KIF4a is overexpressed in cervical cancer (Narayan <italic>et al. </italic>[##REF##17243165##39##]). Treatment of lung cancer cells with siRNA for KIF4A suppressed the growth of these cells and induction of exogenous expression conferred cellular invasive activity on mammalians cells (Taniwaki <italic>et al. </italic>[##REF##18006763##40##]). TUBB3 over-expression is a marker of poor prognostic in pancreatic, non-small cell lung carcinoma and ovarian cancer (Ferrandina <italic>et al. </italic>[##REF##16675570##41##], Seve <italic>et al. </italic>[##REF##16061864##42##,##REF##17289895##43##], Lee <italic>et al. </italic>[##REF##17714470##44##]) and is a potential marker of resistance to treatment by paclitaxel or vinorelbine-based chemotherapy. Kinesin related genes are currently vigorously pursued as therapeutic targets. Taxanes and Vinca alkaloids are major drugs in the clinical routine and are designed to bind and inhibit the motor function of the microtubules. The high frequency of side-effect of these drugs is due to the simultaneous alteration of microtubule cell-resting and differentiation function.</p> <p>Drug companies are now putting an effort into the development of new molecules specifically targeting the kinesin family. Four other genes also present in the signature, TENC1, SEMA3G, CXCL12, and CX3CR1 also play a role in regulating cell motility and migration independently of the microtubule pathways (Chen <italic>et al. </italic>[##REF##11792844##45##], Yu et al [##REF##15882579##46##], Lopez-Bendito <italic>et al. </italic>[##REF##18272682##47##], Lauro <italic>et al. </italic>[##REF##17114429##48##]).</p> <p>Taken together, this shows that many of the genes in the consensus signature have been characterized within a breast cancer con- text, and some have even been considered as drug targets. Nevertheless, not all of them have been covered, opening up possibilities for follow-up analysis. The fact that these genes have been derived from a compendium of six pooled datasets boosts confidence in their relevance within a clinical context.</p>
<title>Results and discussion</title> <title>Resampling on a single breast cancer dataset</title> <p>We applied the repeated random resampling strategy (see Methods section) on the Veer <italic>et al. </italic>[##REF##11823860##1##] dataset, see Figure ##FIG##0##1##. When comparing the frequency-based ranking to the set of genes in the published 70-gene and 231-gene signatures, there appear to be additional genes with high frequencies of occurrence that were not part of the published signatures. Both Michiels <italic>et al. </italic>[##UREF##0##5##], and Roepman <italic>et al. </italic>[##REF##16489042##13##] point out that the genes with high frequencies of occurrence, which are not part of the original signature, are of high interest. They hypothesize that these genes were probably not picked up in the signatures due to sampling specific biases in the dataset. This suggests a beneficial effect due to the resampling which does pick them up. However, since the true ranking of genes underlying breast cancer outcome is unknown, a plausible alternative hypothesis could be that those genes are false positives.</p> <p>Roepman <italic>et al. </italic>[##REF##16489042##13##] considers all genes with a non-zero frequency of occurrence to represent a robust signature. This seems rather optimistic, since genes will be expected to have a positive frequency by chance. Therefore, we propose to set a threshold to prune the set of genes to only those with a frequency higher than expected by chance (see Methods section). This threshold is indicated by the vertical red line in Figure ##FIG##0##1##. The set of genes retained after this pruning step still contains all genes from the original signatures, i.e. all light-blue bars (the original signatures) reside on the left of the threshold. Of course, the ranking itself remains unchanged.</p> <p>In the current experimental setting, it is impossible to conclude which hypothesis correctly describes the appearance of genes with a high (above the threshold) frequency of occurrence, that were not in the original signature.</p> <title>Resampling on a single artificial dataset</title> <p>We set up an experiment on artificial data to be able to compare the ranking after resampling to a known ground truth (see Methods section). The artificial dataset mimics a microarray dataset, and consists of 5000 genes, of which either 500, 2500 or 5000 genes were informative. The experiments were done using artificial datasets consisting of 100 to 1000 samples, with equal priors for the two classes. The data was sampled from a class-conditional gaussian distribution, without any covariance structure. Since genes are assumed to be independent, these datasets are more ideal than real microarray datasets. Nevertheless, this controlled, artificial setting can be a useful tool to gain insight into the relationship between sample size and gene selection.</p> <p>On these artificial datasets we compare the ranking resulting from the resampling strategy with the ranking based on all samples. Additionally, we also consider two variants of the resampling strategy. Instead of only storing the top <italic>N </italic>genes in each of the resamplings, we sum the statistics (sum of ranks or sum of signal-to-noise ratios (SNRs)) of the genes across all the resamplings, and base the final ranking on this cumulative statistic. Summing the statistics amounts to a 'meta-analysis', and the Top <italic>N </italic>method is a 'combining decisions' approach. We then compare the ranking obtained by each method to the ground truth (Spearman rank correlation). Since a particular initialization of the artificial data might influence the results, we repeated the entire experiment 100 times, and present statistics across all repeats (average and standard deviation). Figure ##FIG##1##2## shows the Spearman rank correlation between the known ranks of the artificial data and those obtained by the Top <italic>N </italic>resampling approaches, the two resampling variants, and the ranking over all samples. For <italic>N </italic>we considered values of 50, 100 and 200, similar to Michiels <italic>et al. </italic>[##UREF##0##5##], and Roepman <italic>et al. </italic>[##REF##16489042##13##].</p> <p>Figure ##FIG##1##2## indicates that the ranking based on all samples outperforms the ranking obtained from resampling and then determining the frequency of being in the top <italic>N</italic>. Furthermore, it is evident that combining the statistics (sum of ranks or sum of SNRs) of each resampling performs similar to the ranking over all samples. In fact, it is quite straight-forward to show analytically that, for a given gene, the summation of SNRs across resamplings is equal to the SNR over all samples (see Additional file ##SUPPL##0##1##). This is true when enough resamplings are performed, and they are uniformly sampled from the complete dataset.</p> <p>For smaller artificial datasets the obtained Spearman rank correlations are lower compared to larger artificial datasets. The reduced sample size is the most plausible explanation for this effect. For larger artificial datasets, the gap in performance between the top <italic>N </italic>resampling approaches and the sum of ranks and sum of SNRs widens, indicating that the latter two methods profit more from an increase in artificial dataset size.</p> <p>Taken together, these results clearly show that the random resampling strategy produces a suboptimal ranking of the prognostic genes, and that it is preferable to employ a ranking based on all samples in the dataset. In addition, these differences become more pronounced as the sample size increases. Given that these effects are already evident on an artificial dataset, we hypothesize that the sample size effects will be even stronger on data with a complex covariance structure. We therefore conclude that it is unwise to use the repeated random resampling approach on real data for ranking features.</p> <title>Pooling artificial datasets</title> <p>The experiment on a single artificial dataset is easily extendible to a scenario where multiple artificial datasets are available. Such a setup would emulate the availability of several microarray datasets from the same biological context. We hypothesize that a significant association exists between the classification error and the number of datasets pooled.</p> <p>We set up an experiment using six artificial datasets, each with 200 samples and 5000 genes, of which 500 are informative. Each of these six artificial datasets were generated using the same model, and no additional noise or heterogeneity was introduced. This represents the ideal case, where each of the six datasets are truly relevant for the same underlying problem. We applied the double loop cross validation (DLCV, nearest mean classifier) protocol on each artificial dataset separately (see Methods section), and each possible pooled subset of artificial datasets. First of all, we inspected the behavior of pooling pairs of artificial datasets relative to the performance on the individual artificial datasets. A DLCV error which is lower for the pooled pair compared to the individual errors implies a synergetic effect on the classification performance. When the pooled DLCV error is between the two individual errors, and lower/higher than the weighted mean (number of samples chosen as weights) we label the effect as marginal synergetic/marginal anti-synergetic. Conversely, if the DLCV error is strictly higher than the individual errors it is labeled as anti-synergetic. Figure ##FIG##2##3## shows a network indicating the type of synergy that was observed for the six artificial datasets, labeled Art1 to Art6.</p> <p>Figure ##FIG##2##3## shows that there is a synergetic effect on the DLCV error for 14 of the 15 pooled pairs of artificial datasets. A marginal anti-synergetic effect is only observed in one of the 15 cases. This suggests a large gain in statistical power when pooling the artificial datasets.</p> <p>Next, we evaluated the DLCV error for every potential pooling of one to six artificial datasets. In addition, we evaluated the performance of each of the classifiers derived from the pooled data on an independent large validation set of 2000 artificial samples, see Figure ##FIG##3##4A## and ##FIG##3##4B##. Figure ##FIG##3##4A## and ##FIG##3##4B## show that a high degree of synergy is obtained by pooling artificial datasets. Both the DLCV error as well as the error on the validation set show a highly significant correlation with the number of artificial datasets that is pooled, Pearson correlation of -0.91 (<italic>p </italic>= 1.1<italic>e </italic>- 24) and -0.95 (<italic>p </italic>= 1.4<italic>e </italic>- 31), respectively.</p> <p>To test whether the observed trends are classifier specific, we repeated the experiment using non-linear classifiers (as opposed to the linear nearest mean classifier used above). Thus, we repeated the experiment using a K-Nearest Neighbor classifier (K-NN) as well as a Support Vector Machine with a Radial Basis Function as kernel (SVM with RBF kernel). Using these classifiers, we observed the same trends for the DLCV and Validation error. That is, in all cases the performance gets better after pooling the artificial datasets (trends significant, <italic>p </italic>&lt; 0.05). However, the errors obtained are higher than those from the NMC. The plots are shown in Additional files ##SUPPL##1##2## and ##SUPPL##2##3##.</p> <p>In addition to the classification performance, we looked at the optimal number of genes that is selected by the classifiers, see Figure ##FIG##3##4C##. It turns out that there is a highly significant positive trend (Pearson correlation 0.85, <italic>p </italic>= 1.8<italic>e </italic>- 18) towards selecting more genes when artificial datasets are pooled.</p> <p>In summary, pooling artificial datasets shows that larger training sets result in larger gene sets being selected and, most importantly, in better classification performance. Since this effect is already clearly present in the simple model, we believe that this effect will be even stronger in more complex models which include, for example, a covariance structure between genes. This stems from the fact that these more complex models will require more data (relative to the simple models) to accurately estimate the required complex classifiers.</p> <title>Pooling breast cancer datasets</title> <p>We used a compendium of six Affymetrix breast cancer datasets to evaluate the effect of pooling datasets on the performance of classifiers (see Table ##TAB##0##1## and methods section). Our aim is to classify patients into the poor/good outcome groups as well as possible. To this end, we followed the same strategy as on the artificial datasets. First, we inspected the synergy effect on the DLCV error achieved on pairs of datasets relative to the DLCV errors on the individual datasets (DLCV, nearest mean classifier). Figure ##FIG##4##5## shows a network indicating the synergy among the six real datasets.</p> <p>Figure ##FIG##4##5## shows that for 11 of the 15 pairwise pooled datasets a synergetic effect was observed (green/blue link). For 4 of the 15 pairwise pooled datasets there was a marginal anti-synergetic effect (yellow link), and for none of the pairs the effect was anti-synergetic (orange link).</p> <p>As a next step we applied the DLCV framework on each potential pooling of the datasets, and assessed the DLCV error. In addition, we evaluated the performance of each classifier on a seventh independent dataset (Vijver <italic>et al. </italic>[##REF##12490681##3##]). Figure ##FIG##5##6A## and ##FIG##5##6B## show the DLCV error and validation error as a function of the number of datasets pooled.</p> <p>There is a clear trend indicating that the DLCV error decreases when more datasets are pooled (see Figure ##FIG##5##6A##). The observed trend has a Pearson correlation of -0.26, which is significant: <italic>p </italic>= 4.0<italic>e </italic>- 2. Of course, since the different datasets have different numbers of samples, an alternative would be to evaluate the correlation of the error and the actual number of samples in the pooled datasets. Doing this unveils a similar significant correlation (Additional file ##SUPPL##3##4##).</p> <p>The results on the seventh independent validation set (Vijver <italic>et al. </italic>[##REF##12490681##3##] data) confirm the synergetic effect obtained by pooling datasets (see Figure ##FIG##5##6B##). The trend is even more pronounced at a Pearson correlation of -0.35, with <italic>p </italic>= 5.0<italic>e </italic>- 3. In addition, in a multivariate Cox proportional hazards model the signature is an independent predictor of outcome in the presence of the standard clinico-pathological variables (data not shown). Similarly, plotting the results in a Kaplan-Meier plot indicates a highly significant separation (<italic>p </italic>= 9.5<italic>e </italic>- 7, logrank test, data not shown).</p> <p>The nearest mean classifier that was used in the above experiment is a relatively simple linear classifier, which is known to perform well on this type of data [##REF##15817694##26##]. To test whether the observed trend is classifier specific, we repeated the experiment using non-linear classifiers. Thus, we repeated the experiment using a K-Nearest Neighbor classifier (K-NN) as well as a Support Vector Machine with a Radial Basis Function as kernel (SVM with RBF kernel). The results that were obtained for the DLCV error and the validation error on the Vijver dataset are shown in Additional files ##SUPPL##4##5## and ##SUPPL##5##6##, for the K-NN and SVM classifiers, respectively.</p> <p>Overall, the average eFPFN across all 63 different pooling combinations when using the K-NN classifier (DLCV = 0.356) is slightly worse compared to the nearest mean classifier (DLCV = 0.348). In addition, the error obtained using the SVM (DLCV = 0.391) is higher than the K-NN. Similarly, the average error on the independent validation set (Vijver) is also higher: (0.329 for the NMC, 0.365 for the K-NN and 0.405 for the SVM). This corresponds to the previous observation that simpler classifier often perform better on this type of data. At the same time, the trend that classifiers that are derived from pooled datasets are more accurate is also observed.</p> <p>In conclusion, not only simple linear classifiers, but also more complex non-linear classifiers show a clear benefit by training them on pooled datasets. Moreover, these results confirm that the linear nearest mean classifier is the best candidate for this type of classification problem.</p> <p>Previously reported classifiers have been using varying numbers of genes. Signatures that have been derived using a cross validation setup that is similar to the one used here, use 52 [##REF##17076897##20##] to around 70 genes [##REF##15721472##2##,##REF##12490681##3##]. We have investigated the number of genes selected by the DLCV method, for each of our pooled datasets, see Figure ##FIG##5##6C##. The classifiers trained on a single dataset use around 50 genes, whereas pooled datasets select progressively more genes. Especially the classifiers from five or six datasets select more than 100 genes. These results indicate a positive trend in the number of datasets pooled and the number of genes selected in the signatures (Pearson correlation of 0.38 with <italic>p </italic>= 1.9<italic>e </italic>- 3).</p> <p>In summary, the effects observed when pooling artificial datasets are also observed when pooling breast cancer datasets. This implies that datasets should be analyzed jointly rather than in isolation.</p> <title>Enrichment of the signatures</title> <p>Functional enrichment of prognostic signatures has unveiled a multitude of pathways that are associated with tumor progression. We set out to inspect the effect of pooling datasets on the enrichment. We used a collection of 1860 genes sets consisting of at least five annotated genes (see Methods section). For each pooled combination, we used the hypergeometric test to assess the significance of the overlap, followed by Bonferroni correction (see Methods section).</p> <p>The most highly enriched categories were proliferation associated, which are well-known to be associated with tumorigenesis. Figure ##FIG##6##7## shows the enrichment for three of the gene sets. Figure ##FIG##7##8## depicts a complete heatmap of enrichments.</p> <p>As seen in Figure ##FIG##6##7A##, the mitosis GO category (GO:0007067: mitosis) shows a significant enrichment for practically every pooled combination of datasets. This is an example of a very strong signal, which gets even stronger when pooling datasets. Another proliferation related category (Figure ##FIG##6##7B##), the cell cycle pathway (KEGG – hsa04110 – Cell cycle) becomes significantly enriched when roughly three or more datasets are combined. Thus it's association is weaker than the mitosis gene set, but gets detected when pooling a few datasets.</p> <p>Lastly, a gene set associated to microtubule activity (GO:0003777: microtubule motor activity) is non-significant for the range of pooling one to three datasets (Figure ##FIG##6##7C##). When combining four or five it is sometimes picked up, and is definitely picked up when pooling six datasets. This is a clear example of a relatively weak signal, which benefits significantly from pooling datasets.</p> <p>Conversely, we also observe that many gene sets which are significant in some of the single/pairs of datasets are no longer significant when pooling (Figure ##FIG##7##8##). For instance, glycolysis related pathways are only picked up in the signature extracted from the Pawitan <italic>et al. </italic>[##REF##16280042##23##] dataset. This is evidence that those categories are specific to this dataset, but represent false positives within the context of the global breast cancer population.</p> <p>Overall, the enrichment analysis provides further proof that pooling datasets significantly increases the statistical power to detect weak signals and filter out false positive associations.</p> <title>Heterogeneity amongst the breast cancer datasets</title> <p>Pooling datasets with a high degree of heterogeneity might have a detrimental effect on subsequent analysis. A potential way to limit the heterogeneity would be to restrict the analysis to known clinical subgroups. To test this, we split each dataset based on ER status, and repeated the analysis on each of these subgroups. Of course, these splits reduce the number of samples and intensify small sample size related problems. For this reason it becomes impractical to exhaustively explore all possible stratifications of the dataset. We chose the ER status since it is known to have a profound effect on disease progression and gene expression.</p> <p>On the subgroup of ER positive samples, we observed the same trends as on the complete datasets. We observed a synergetic effect for 10 of the 15 of the pooled pairs of datasets (see Additional file ##SUPPL##6##7##). Moreover, the trends observed on all samples are also significant on the ER positive subset (see Additional file ##SUPPL##7##8##). Enrichment analysis of these signatures revealed the same set of categories that were found on the complete set of samples (see Additional file ##SUPPL##8##9##).</p> <p>Similarly, we also repeated all analyses on the ER negative subgroup of samples. We observe the same trends as on all samples or the ER positive group (see Additional file ##SUPPL##9##10## and ##SUPPL##10##11##). However, the classification accuracy on this subgroup is much lower. A potential explanation for this observation is that this group of samples is inherently more difficult to classify, or that the small sample size related problems play a big role (combining all six datasets results in 191 samples).</p> <p>Enrichment analysis of the signatures from the ER negative group, leads to a very limited set of enriched gene sets (see Additional file ##SUPPL##11##12##). The most highly enriched category is 'GO:0006955: immune response'. This confirms previous reports on the association of this category with outcome in the ER negative group [##REF##17683518##27##,##REF##17910759##28##]. However, in the largest dataset (all ER negative samples), not a single category is enriched. In both of the ER groups, we observed the same synergetic trends when pooling datasets on cross validation, independent validation, and the number of genes selected. Therefore, heterogeneity among the datasets due to ER status is only limited.</p> <title>Overlap amongst signatures</title> <p>It has been previously observed that existing signatures have limited overlap in terms of genes. Several potential explanations have been proposed for these differences:</p> <p>1. Heterogeneity in expression due to different platform technologies and references;</p> <p>2. Differences in supervised protocols with which signatures are extracted;</p> <p>3. Although the genes are not exactly the same, they represent the same set of pathways;</p> <p>4. Differences in clinical composition between datasets (i.e. sample heterogeneity) and</p> <p>5. Small sample size problems that cause inaccurate signatures.</p> <p>Here we will critically evaluate whether we observe support for each of these explanations in the context of the six breast cancer datasets we studied.</p> <title>Explanations 1 and 2</title> <p>In our analysis, we have restricted ourselves to the same platform for training our classifiers, and use the same supervised protocol on all datasets. This allowed us to check whether the first two explanations apply. We evaluated the relative overlap of every combination of two signatures that were extracted from a single dataset, see Figure ##FIG##8##9A##. For these 15 comparisons, the majority of genes are exclusively part of one signature (mean overlap 1.8%). This clearly indicates that, within the context of the datasets studied here, Explanations 1 and 2 are not the likely causes of the limited overlap between the signatures from these six datasets.</p> <title>Explanation 3</title> <p>Using our functional enrichment analysis we checked Explanation 3. The enrichment patterns of the signatures from single datasets are largely disparate (Figure ##FIG##6##7##). Two of the six signatures are not enriched for any functional category. There is only one gene set (GO:0007049: cell cycle), which is enriched in three signatures, namely those derived from Desmedt <italic>et al. </italic>[##REF##17545524##4##], Miller <italic>et al. </italic>[##REF##16141321##18##], and Loi <italic>et al. </italic>[##REF##17401012##24##]. When considering the set of enriched gene sets that overlap between every pair of signatures (Figure ##FIG##8##9D##), we observed a mean overlap of less than 1%. Taken together, this indicates that there is very little support for the third explanation in these six datasets.</p> <title>Explanation 4</title> <p>Breast cancer is well known to be a complex disease which manifests itself in a range of subtypes. As a result, differences in the clinical composition of different datasets might explain the different signatures. However, when creating a histogram of signature overlap within the ER positive/negative subgroups we also found that the majority of genes are only part of one signature. Thus, heterogeneity with respect to ER status does not explain the limited degree of overlap among the signatures from these six datasets. Of course, it could still be that differences in the composition with respect to another (unknown) clinical parameter partly explain the lack of overlap.</p> <title>Explanation 5</title> <p>Signatures that are derived from pooled datasets have been derived using more samples, thereby easing small sample size related difficulties. Therefore, we inspected the relative overlap between signatures derived from one, two and three datasets. To avoid any bias, we only consider pairs of pooled datasets that do not overlap in terms of samples. The histogram and p-values shown in Figure ##FIG##8##9A–C## represent the mean over all these combinations. The overlap becomes progressively larger for signatures that are extracted from larger datasets (single dataset: 1.8%, two pooled datasets: 8.1%, three pooled datasets: 13.1%). At the same time this overlap becomes more significant (single dataset: -<italic>log</italic>10(<italic>p</italic>) = 2.1, two pooled datasets: -<italic>log</italic>10(<italic>p</italic>) = 13.6, three pooled datasets: -<italic>log</italic>10(<italic>p</italic>) = 23.2). The same trends were observed on the artificial data, and the ER positive subgroup (results not shown). Next, we repeated this analysis by looking for overlap in terms of enriched gene sets (Figure ##FIG##8##9D–F##). Similarly, signatures derived from the pooled datasets show a higher degree of enriched gene sets that overlap. The distinguishing property of these signatures is the fact that they were derived from a larger dataset, i.e. more samples. Thus, we conclude that Explanation 5 is the most relevant explanation for the low degree of overlap in genes from signatures derived from these six datasets.</p> <title>Consensus signature across six datasets</title> <p>The signature from the six pooled datasets will be substantially different from those derived from each individual dataset. To gain insight into the difference, we created a chart indicating the rank position of all 127 genes selected in the classifier trained on all six datasets, along with rank positions from these genes in the rankings obtained from the individual datasets (Figure ##FIG##9##10##). Additional file ##SUPPL##12##13## contains detailed information on the 127 genes, and the centroids of the classifier. Most of the 127 signature genes, had a much lower rank (larger rank number) in the individual signatures, implying that they're not part of those signatures. The difference in ranking of the 127 genes is especially large compared to the signature derived from the Minn <italic>et al. </italic>[##REF##16049480##22##] and Chin <italic>et al. </italic>[##REF##17157792##25##] data. This implies that those rankings are dominated by genes which are not generally relevant for breast cancer.</p> <p>We compared our signature to signatures derived from different breast cancer datasets. We compared to the 70-gene signature from Wang <italic>et al. </italic>[##REF##15721472##2##], 76-gene signature from Vijver <italic>et al. </italic>[##REF##12490681##3##], 70-gene signature from Naderi <italic>et al. </italic>[##REF##16936776##19##], and the 52-gene signature from Teschendorff <italic>et al. </italic>[##REF##17076897##20##]. The latter of which was derived from the first three datasets. None of these three datasets were employed to derive the 127 gene signature. The overlap between the 127 genes and the Teschendorff signature is more significant (<italic>p </italic>= 2.3<italic>e </italic>- 18) than with Naderi (<italic>p </italic>= 1.2<italic>e </italic>- 11), Vijver (<italic>p </italic>= 7.4<italic>e </italic>- 7) or Wang (<italic>p </italic>= 4.7<italic>e </italic>- 1). This provides additional evidence that the limited signature overlap is most likely caused by small sample size difficulties.</p> <p>The set of genes in the signature from all six datasets contains many well known and described genes. For instance, it confirms the significance of HMMR (Rank 127, Figure ##FIG##9##10##) which was only recently associated with breast cancer [##REF##17922014##29##]. This gene was not picked up in any of the individual signatures. In the same study, Pujana <italic>et al. </italic>[##REF##17922014##29##] derived a set of 164 genes that are associated with BRCA1, BRCA2, ATM, and CHEK2. This set of 164 genes is significantly overrepresented in the 127 signature genes (<italic>p </italic>= 2.4<italic>e </italic>- 9).</p> <title>Biological relevance of the consensus signature</title> <p>Cell cycle and Mitosis pathways are the common background of most prognostic gene expression classifiers. Not surprisingly, many of the 127 genes have been previously described as markers of proliferation and poor prognosis in different types of cancer. These include: BUB1B, BUB1, MAD2L1, C16orf61 (DC13), CCT5, EIF4EBP1, MELK, MYBL2, CCNA2, CCNB1 and CCNB2. More specifically, EIF4EBP1 (4E-binding protein 1, 4E-BP1) is a EIF4E-binding protein that plays a critical role in the control of protein synthesis, survival, and cell growth. Rojo <italic>et al. </italic>[##REF##17200342##30##] showed that 4EBP1 is activated in a high percentage of breast tumors, and is associated with higher malignant grade, tumor size, and local recurrence regardless of HER2/neu status (Armengol <italic>et al. </italic>[##REF##17699757##31##]). Lin <italic>et al. </italic>[##REF##17280616##32##] showed that MELK (maternal embryonic leucine zipper kinase), is overexpressed in both breast cancer specimens and cancer cell lines, and that its kinase activity possibly plays a significant role in breast cancer cell proliferation. They show that down-regulation of MELK by treatment with siRNA significantly suppressed the cell growth of breast cancer, indicating its crucial role in the proliferation of breast cancer.</p> <p>C16orf61 (DC13) and CCT5 are examples of genes associated with therapy resistance. More specifically, C16orf61 (DC13) was previously found to be involved in multidrug resistance in breast cancer [##REF##17659439##33##], and mRNA levels of CCT5 were significantly associated with resistance to docetaxel [##REF##16821082##34##].</p> <p>Cellular movement and cytokinesis pathways are significantly over-represented in our prognostic signature mainly through genes (KIF23, KIFC1, KIF14, KIF20A, KIF2C, KIF13B, KIF4A, PRC1, TUBB3, TPX2, CENP-A) linked to the microtubule motor activity (GO:0003777) and the microtubule-based movement (GO:0007018) GO categories. It is of major interest that these GO categories are non-significant for the range of pooling one to three datasets and are only picked up when pooling more datasets.</p> <p>The genes that are associated with the microtubule categories are all part of the kinesin family. This family is composed of more than 40 genes involved in spindle assembly and function, chromosome segregation, mitotic checkpoint control and cytokinesis. They are motor proteins that generate movement along microtubules. KIF14 is a prognostic factor in breast and lung cancer (Corson <italic>et al. </italic>[##REF##16570270##35##,##REF##17545527##36##]). KIF20A is over-expressed in pancreatic ductal adenocarcinoma (PDAC) and siRNA knockdown of KIF20A expression in PDAC cell lines attenuated growth of those cells (Taniuchi <italic>et al. </italic>[##REF##15665285##37##]). KIF2C is also known to be over-expressed in breast cancer cells. Treatment of breast cancer cells (T47D and HBC5) by siRNA against KIF2C suppressed KIF2C expression and inhibited the growth of the breast cancer cell lines (Shimo <italic>et al. </italic>[##REF##17944972##38##]). Immunoprecipitation assay showed an interaction between PRC1 and KIF2C in breast cancer cells (Shimo <italic>et al. </italic>[##REF##17944972##38##]). KIF4a is overexpressed in cervical cancer (Narayan <italic>et al. </italic>[##REF##17243165##39##]). Treatment of lung cancer cells with siRNA for KIF4A suppressed the growth of these cells and induction of exogenous expression conferred cellular invasive activity on mammalians cells (Taniwaki <italic>et al. </italic>[##REF##18006763##40##]). TUBB3 over-expression is a marker of poor prognostic in pancreatic, non-small cell lung carcinoma and ovarian cancer (Ferrandina <italic>et al. </italic>[##REF##16675570##41##], Seve <italic>et al. </italic>[##REF##16061864##42##,##REF##17289895##43##], Lee <italic>et al. </italic>[##REF##17714470##44##]) and is a potential marker of resistance to treatment by paclitaxel or vinorelbine-based chemotherapy. Kinesin related genes are currently vigorously pursued as therapeutic targets. Taxanes and Vinca alkaloids are major drugs in the clinical routine and are designed to bind and inhibit the motor function of the microtubules. The high frequency of side-effect of these drugs is due to the simultaneous alteration of microtubule cell-resting and differentiation function.</p> <p>Drug companies are now putting an effort into the development of new molecules specifically targeting the kinesin family. Four other genes also present in the signature, TENC1, SEMA3G, CXCL12, and CX3CR1 also play a role in regulating cell motility and migration independently of the microtubule pathways (Chen <italic>et al. </italic>[##REF##11792844##45##], Yu et al [##REF##15882579##46##], Lopez-Bendito <italic>et al. </italic>[##REF##18272682##47##], Lauro <italic>et al. </italic>[##REF##17114429##48##]).</p> <p>Taken together, this shows that many of the genes in the consensus signature have been characterized within a breast cancer con- text, and some have even been considered as drug targets. Nevertheless, not all of them have been covered, opening up possibilities for follow-up analysis. The fact that these genes have been derived from a compendium of six pooled datasets boosts confidence in their relevance within a clinical context.</p>
<title>Conclusion</title> <p>In our analysis we aimed to extract a robust classifier from multiple datasets. We explored the possibility of pooling datasets, which should alleviate small sample size related problems, and increase robustness. For a single dataset, we revisited the resampling strategy [##UREF##0##5##,##REF##16489042##13##] which has been employed as a 'gold standard' for evaluating gene sets and has been applied to extract robust signatures, e.g. Yu <italic>et al. </italic>[##REF##17894856##14##]. Our analysis provides proof that the resampling strategy does not alleviate potential problems due to sampling biases. Instead, by only taking the top <italic>N </italic>genes in every resampling into account, the obtained feature ranking is inferior to the ranking obtained based on all samples. Combining the statistics from each of the individual resamplings (either SNRs or Ranks), provides a ranking which is equivalent to the ranking based on all samples. Therefore, we propose to use the ranking over all samples in order to train a final classifier. </p> <p>Concurrent analysis of multiple datasets is an established way to increase statistical power, and thereby robustness. Pooling the datasets prior to any analysis would enable the largest gain in statistical power. We presented an example of pooling artificial datasets, which unveiled a significant negative correlation between the number of datasets which is pooled, and the DLCV error/validation error. We also observed a significant positive correlation between the degree of pooling and the number of genes which is selected. Thus, on the artificial data there is a clear synergetic effect associated with pooling of datasets.</p> <p>Similarly, pooling breast cancer datasets represents a way to improve classification performance, since the pooled datasets will provide a better resemblance to the true underlying breast cancer population. For instance, breast cancer subtypes that are not present in one dataset might be present in another. A classifier trained on a pooled dataset was hypothesized to be more robust when applied to unseen, independent data. To test this hypothesis, we used a collection of six independent datasets, all measured on the same platform. Indeed, we observed a lower classification error for classifiers trained on pooled datasets. This significant trend is supported by a validation on a seventh independent dataset. The trends observed for pooling datasets strongly confirm the belief that there is a synergetic effect between pooling datasets and classification performance. Recently, another group [##REF##18304324##21##] has come to a similar conclusion, albeit using only three datasets for training, and one for testing. A shortcoming in their analysis is that they do not switch the roles of which datasets are used for training and testing. Moreover, they mention the issue of small overlap among signatures, but do not provide any clear answers to that issue. Pooling in itself might lead to unwanted effects (i.e. Simpson's Paradox, [##REF##11937652##49##]). Our results do not rule out the existence of detrimental effects due to pooling heterogeneous datasets, but imply that the synergetic effect of pooling is stronger than the detrimental effect of heterogeneity.</p> <p>Signatures that have been extracted from independent cohorts of patients share only a limited number of genes. A variety of explanations have been proposed, only some of which have been thoroughly tested. We have checked whether there is any support for five of these explanations. On a collection of six breast cancer datasets, which were all measured on the same array, we applied the same supervised protocol. This lead to signatures that still lack overlap in terms of genes contained in the signatures. Similarly, our results point out that the overlap in terms of enriched pathways is just as limited, and reducing heterogeneity by stratifying based on ER status does not resolve the issue either. Of course, ER status might not be the only important source of heterogeneity. Therefore, we cannot completely rule out the effect of data heterogeneity on signature overlap. We believe that the limited signature overlap is most likely due to small sample size problems (Explanation 5). More specifically, that the variability between classifiers stems from the fact that the individual datasets are too small to capture the important sources of variation in the data. Moreover, pooling datasets might average-out adverse effects from heterogeneity that is only present in a subset of the samples/datasets. Explanation 5 is supported by a much larger overlap between signatures from pooled datasets than between signatures derived from single datasets. Nevertheless, we would like to point out that these conclusions might be specific to the six breast cancer datasets analyzed here and the specific techniques we employed to perform these analyses. That is, this conclusion may not extrapolate to different datasets/other tumor types.</p> <p>Further evidence for the synergetic effect of pooling datasets comes from the functional enrichment analysis. We observed both low signal associations that become stronger, as well as categories that we can label as false positive associations since they're only found in one dataset and disappear when the complete set is analyzed. Moreover, we pick up relevant breast cancer associated genes in the signature derived from six pooled dataset, that are not picked up in any of the signatures from the separate datasets. Many of the genes in the signature have been previously characterized as drug targets and since our analysis is based on a large compendium it boosts our confidence in their potential relevance for clinical practice.</p> <p>Overall, we hypothesize that the 127 gene signature that was derived from the six pooled datasets is one of the most robust signatures currently available. Our results indicate that new breast cancer datasets should not be analyzed in isolation, but should be analyzed in the context of the available compendium of breast cancer samples</p>
<p>This is an Open Access article distributed under the terms of the Creative Commons Attribution License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/2.0"/>), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</p> <title>Background</title> <p>Michiels <italic>et al. </italic>(Lancet 2005; 365: 488–92) employed a resampling strategy to show that the genes identified as predictors of prognosis from resamplings of a single gene expression dataset are highly variable. The genes most frequently identified in the separate resamplings were put forward as a 'gold standard'. On a higher level, breast cancer datasets collected by different institutions can be considered as resamplings from the underlying breast cancer population. The limited overlap between published prognostic signatures confirms the trend of signature instability identified by the resampling strategy. Six breast cancer datasets, totaling 947 samples, all measured on the Affymetrix platform, are currently available. This provides a unique opportunity to employ a substantial dataset to investigate the effects of pooling datasets on classifier accuracy, signature stability and enrichment of functional categories.</p> <title>Results</title> <p>We show that the resampling strategy produces a suboptimal ranking of genes, which can not be considered to be a 'gold standard'. When pooling breast cancer datasets, we observed a synergetic effect on the classification performance in 73% of the cases. We also observe a significant positive correlation between the number of datasets that is pooled, the validation performance, the number of genes selected, and the enrichment of specific functional categories. In addition, we have evaluated the support for five explanations that have been postulated for the limited overlap of signatures.</p> <title>Conclusion</title> <p>The limited overlap of current signature genes can be attributed to small sample size. Pooling datasets results in more accurate classification and a convergence of signature genes. We therefore advocate the analysis of new data within the context of a compendium, rather than analysis in isolation.</p>
<title>Authors' contributions</title> <p>MHvV, LFAW, and MJTR contributed to the conceptual design of the study. MHvV and FR performed the analysis. MHvV, FR, HMH, and LFAW wrote the manuscript. HMH and MJvdV participated in the discussion of the results.</p> <title>Supplementary Material</title>
<title>Acknowledgements</title> <p>We thank A. Wirapati for valuable discussions on the work presented. This work was part of the BioRange programme of the Netherlands Bioinformatics Centre (NBIC), which is supported by a BSIK grant through the Netherlands Genomics Initiative (NGI).</p>
<fig position="float" id="F1"><label>Figure 1</label><caption><p><bold>Result of the repeated random resampling procedure on the Veer <italic>et al.</italic></bold>[##REF##11823860##1##]<bold> data.</bold> The histogram shows the frequencies of genes being among the top 200 genes over 500 resamplings. Below the histogram, two lanes containing light-blue bars indicate the genes that are part of the published signatures. The red line indicates the frequency threshold corresponding to the expected value of the frequency under the null hypothesis (no information) given the number of genes <italic>N </italic>that is selected in each of the <italic>R </italic>resamplings.</p></caption></fig> <fig position="float" id="F2"><label>Figure 2</label><caption><p><bold>Spearman rank correlation of the ranking obtained after resampling an artificial dataset and the true ranking.</bold> The number of informative genes was varied from 500 (A and D), 2500 (B and E), to 5000 (C and F), and 100 (A, B and C) or 500 (D, E and F) resamplings were considered. The errorbars indicate the mean and standard deviation over 100 repeats of the entire experiment. The results from the 'All Samples', 'Sum of Ranks', and 'Sum of SNRs' methods are equivalent, and are therefore plotted on top of each other (top line in all plots).</p></caption></fig> <fig position="float" id="F3"><label>Figure 3</label><caption><p><bold>Network indicating the synergy between six artificial datasets (Art1 to Art6).</bold> Each of these six datasets were generated from the same model, without introducing any noise or heterogeneity. Each node represents a dataset, and each edge the effect on the DLCV error when pooling them. Four different effects were considered, synergy (bright green) when the pooled error is lower than each of the separate errors. Marginal synergy (light blue) when the pooled error is lower than the weighted mean of the separate errors, conversely marginal anti-synergy (yellow) when it is higher. Lastly, true anti-synergy (orange) indicates a higher DLCV error for the pooled dataset.</p></caption></fig> <fig position="float" id="F4"><label>Figure 4</label><caption><p><bold>Scatterplot indicating the classification error relative to the number of datasets that is pooled.</bold> A) DLCV error. B) Error on a large independent validation set of 2000 samples. C) Number of genes selected by the DLCV protocol. The color corresponds to the number of datasets that was used. Poolings with the same number of datasets are sorted based on error/number of genes. Labels indicate which combination of datasets was used.</p></caption></fig> <fig position="float" id="F5"><label>Figure 5</label><caption><p><bold>Network indicating the synergy between six real datasets.</bold> Each node represents a dataset, and each edge the effect on the DLCV error when pooling them. Four different effects were considered, synergy (bright green) when the pooled error is lower than each of the separate errors. Marginal synergy (light blue) when the pooled error is lower than the weighted mean of the separate errors, conversely marginal anti-synergy (yellow) when it is higher. Lastly, true anti-synergy (orange) indicates a higher DLCV error for the pooled dataset.</p></caption></fig> <fig position="float" id="F6"><label>Figure 6</label><caption><p><bold>Scatterplot indicating the classification error relative to the number of datasets that is pooled.</bold> A) DLCV error. B) Error on the Vijver <italic>et al. </italic>[##REF##12490681##3##] dataset. C) Number of genes selected by the DLCV protocol. The color corresponds to the number of datasets that was used. Poolings with the same number of datasets are sorted based on error/number of genes. Labels indicate which combination of datasets was used.</p></caption></fig> <fig position="float" id="F7"><label>Figure 7</label><caption><p><bold>Enrichment of three gene sets relative to the number of datasets which is pooled.</bold> A) GO:0007067: mitosis B) KEGG – hsa04110 – Cell cycle C) GO:0003777: microtubule motor activity. Scatterplots indicate the minus log10 of the Bonferroni corrected p-values. The red line indicates the level at which 0.01 is reached.</p></caption></fig> <fig position="float" id="F8"><label>Figure 8</label><caption><p><bold>Heatmap of the Bonferroni corrected p-values of the enrichment between each signature and a collection of gene sets.</bold> Only categories with at least 1 significant association are shown.</p></caption></fig> <fig position="float" id="F9"><label>Figure 9</label><caption><p><bold>Histograms indicating the percentage of genes (A-C) and enriched gene sets (D-F) that overlap between two signatures.</bold> A and D) Median histogram and hypergeometric p-value across every pairwise comparison of signatures from single datasets. B and E) Median histogram and hypergeometric p-value across every pairwise comparison of signatures from 2 pooled datasets C and F) Average histogram and hypergeometric p-value across every pairwise comparison of signatures from 3 pooled datasets. We only considered the comparisons of pooled datasets that do not overlap in terms of samples, e.g. the comparison of the signatures from 'Des Loi' and 'Des Paw' is excluded to avoid any bias.</p></caption></fig> <fig position="float" id="F10"><label>Figure 10</label><caption><p><bold>Chart listing the 127 genes selected in the classifier trained on all six datasets.</bold> For each gene, we list the rank, Entrez id, and Gene symbol. Green cell shading indicates the genes that are part of the signature from the six pooled datasets, which are not part of any of the signatures from the single datasets. Yellow cell shading indicates the seven microtubule associated genes. The succeeding columns indicate the rank position of a particular gene in each of the six separate rankings. An orange cell shading indicates the genes that were part of the individual signatures. The purple cell shading indicates the overlap to a group of existing breast cancer signatures (Wang <italic>et al. </italic>[##REF##15721472##2##], Vijver <italic>et al. </italic>[##REF##12490681##3##], Naderi <italic>et al. </italic>[##REF##16936776##19##], Teschendorff <italic>et al. </italic>[##REF##17076897##20##]), and a group of breast cancer associated genes (Pujana <italic>et al. </italic>[##REF##17922014##29##]). P-values indicating the significance of the overlap (hypergeometric test) of these signatures is given at the bottom of the columns.</p></caption></fig>
<table-wrap position="float" id="T1"><label>Table 1</label><caption><p>Indication of the origin of the seven datasets that were used.</p></caption><table frame="hsides" rules="groups"><thead><tr><td/><td/><td/><td/><td align="center" colspan="2">All ER</td><td align="center" colspan="2">ER pos</td><td align="center" colspan="2">ER neg</td></tr><tr><td align="left">Publication:</td><td align="left">Label</td><td align="left">Survival</td><td align="right">Total</td><td align="right">Poor</td><td align="right">Good</td><td align="right">Poor</td><td align="right">Good</td><td align="right">Poor</td><td align="right">Good</td></tr></thead><tbody><tr><td align="left">Desmedt <italic>et al. </italic>[##REF##17545524##4##]</td><td align="left">Des (D)</td><td align="left">DMFS</td><td align="right">120</td><td align="right">29</td><td align="right">91</td><td align="right">16</td><td align="right">65</td><td align="right">13</td><td align="right">26</td></tr><tr><td align="left">Minn <italic>et al. </italic>[##REF##16049480##22##]</td><td align="left">Min (Mn)</td><td align="left">DMFS</td><td align="right">62</td><td align="right">21</td><td align="right">41</td><td align="right">9</td><td align="right">30</td><td align="right">12</td><td align="right">11</td></tr><tr><td align="left">Miller <italic>et al. </italic>[##REF##16141321##18##]</td><td align="left">Mil (Ml)</td><td align="left">SOS</td><td align="right">193</td><td align="right">37</td><td align="right">156</td><td align="right">26</td><td align="right">117</td><td align="right">11</td><td align="right">39</td></tr><tr><td align="left">Pawitan <italic>et al. </italic>[##REF##16280042##23##]</td><td align="left">Paw (P)</td><td align="left">SOS</td><td align="right">142</td><td align="right">22</td><td align="right">120</td><td align="right">14</td><td align="right">99</td><td align="right">8</td><td align="right">21</td></tr><tr><td align="left">Loi <italic>et al. </italic>[##REF##17401012##24##]</td><td align="left">Loi (L)</td><td align="left">DMFS</td><td align="right">120</td><td align="right">28</td><td align="right">92</td><td align="right">21</td><td align="right">71</td><td align="right">7</td><td align="right">21</td></tr><tr><td align="left">Chin <italic>et al. </italic>[##REF##17157792##25##]</td><td align="left">Chi (C)</td><td align="left">DMFS</td><td align="right">86</td><td align="right">23</td><td align="right">63</td><td align="right">14</td><td align="right">50</td><td align="right">9</td><td align="right">13</td></tr><tr><td align="left">Vijver <italic>et al. </italic>[##REF##12490681##3##]</td><td align="left">Vij (V)</td><td align="left">DMFS</td><td align="right">248</td><td align="right">70</td><td align="right">178</td><td align="right">44</td><td align="right">149</td><td align="right">26</td><td align="right">29</td></tr></tbody></table></table-wrap>
<inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M1" name="1471-2164-9-375-i1" overflow="scroll"><mml:semantics><mml:mrow><mml:mi>p</mml:mi><mml:mo>=</mml:mo><mml:mfrac><mml:mi>N</mml:mi><mml:mi>G</mml:mi></mml:mfrac></mml:mrow></mml:semantics></mml:math></inline-formula> <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M2" name="1471-2164-9-375-i2" overflow="scroll"><mml:semantics><mml:mrow><mml:mi>E</mml:mi><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:mi>R</mml:mi><mml:mi>N</mml:mi></mml:mrow><mml:mi>G</mml:mi></mml:mfrac></mml:mrow></mml:semantics></mml:math></inline-formula> <disp-formula id="bmcM1"><label>(1)</label><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M3" name="1471-2164-9-375-i3" overflow="scroll"><mml:semantics><mml:mrow><mml:mi>t</mml:mi><mml:mo>=</mml:mo><mml:mfrac><mml:mi>N</mml:mi><mml:mi>G</mml:mi></mml:mfrac><mml:mo>.</mml:mo></mml:mrow></mml:semantics></mml:math></disp-formula> <disp-formula id="bmcM2"><label>(2)</label><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M4" name="1471-2164-9-375-i4" overflow="scroll"><mml:semantics><mml:mrow><mml:mi>S</mml:mi><mml:mi>N</mml:mi><mml:mi>R</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mi>i</mml:mi><mml:mo stretchy="false">)</mml:mo><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:mo>|</mml:mo><mml:mi>μ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mn>1</mml:mn></mml:mrow></mml:msub><mml:mo stretchy="false">)</mml:mo><mml:mo>−</mml:mo><mml:mi>μ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mn>2</mml:mn></mml:mrow></mml:msub><mml:mo stretchy="false">)</mml:mo><mml:mo>|</mml:mo></mml:mrow><mml:mrow><mml:mi>σ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mn>1</mml:mn></mml:mrow></mml:msub><mml:mo stretchy="false">)</mml:mo><mml:mo>+</mml:mo><mml:mi>σ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:msub><mml:mi>A</mml:mi><mml:mrow><mml:mi>i</mml:mi><mml:mn>2</mml:mn></mml:mrow></mml:msub><mml:mo stretchy="false">)</mml:mo></mml:mrow></mml:mfrac></mml:mrow></mml:semantics></mml:math></disp-formula> <disp-formula id="bmcM3"><label>(3)</label><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M5" name="1471-2164-9-375-i5" overflow="scroll"><mml:semantics><mml:mrow><mml:msub><mml:mi>e</mml:mi><mml:mrow><mml:mi>F</mml:mi><mml:mi>P</mml:mi><mml:mi>F</mml:mi><mml:mi>N</mml:mi></mml:mrow></mml:msub><mml:mo>=</mml:mo><mml:mfrac><mml:mrow><mml:mfrac><mml:mrow><mml:mi>F</mml:mi><mml:mi>N</mml:mi></mml:mrow><mml:mrow><mml:mi>T</mml:mi><mml:mi>P</mml:mi><mml:mo>+</mml:mo><mml:mi>F</mml:mi><mml:mi>N</mml:mi></mml:mrow></mml:mfrac><mml:mo>+</mml:mo><mml:mfrac><mml:mrow><mml:mi>F</mml:mi><mml:mi>P</mml:mi></mml:mrow><mml:mrow><mml:mi>F</mml:mi><mml:mi>P</mml:mi><mml:mo>+</mml:mo><mml:mi>T</mml:mi><mml:mi>N</mml:mi></mml:mrow></mml:mfrac></mml:mrow><mml:mn>2</mml:mn></mml:mfrac></mml:mrow></mml:semantics></mml:math></disp-formula> <disp-formula id="bmcM4"><label>(4)</label><italic>P</italic>(<italic>X</italic>|<italic>ω</italic><sub>1</sub>) = <italic>N</italic>(<italic>μ</italic>(<italic>j</italic>), 1)</disp-formula> <disp-formula id="bmcM5"><label>(5)</label><italic>P</italic>(<italic>X</italic>|<italic>ω</italic><sub>2</sub>) = <italic>N</italic>(-<italic>μ</italic>(<italic>j</italic>), 1)</disp-formula> <disp-formula id="bmcM6"><label>(6)</label><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M6" name="1471-2164-9-375-i6" overflow="scroll"><mml:semantics><mml:mrow><mml:mi>μ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mi>j</mml:mi><mml:mo stretchy="false">)</mml:mo><mml:mo>=</mml:mo><mml:mrow><mml:mo>{</mml:mo><mml:mrow><mml:mtable columnalign="left"><mml:mtr columnalign="left"><mml:mtd columnalign="left"><mml:mrow><mml:msub><mml:mi>μ</mml:mi><mml:mn>0</mml:mn></mml:msub><mml:mo>−</mml:mo><mml:mfrac><mml:mrow><mml:msub><mml:mi>μ</mml:mi><mml:mn>0</mml:mn></mml:msub><mml:mo stretchy="false">(</mml:mo><mml:mi>j</mml:mi><mml:mo>−</mml:mo><mml:mn>1</mml:mn><mml:mo stretchy="false">)</mml:mo></mml:mrow><mml:mi>J</mml:mi></mml:mfrac></mml:mrow></mml:mtd><mml:mtd columnalign="left"><mml:mrow><mml:mtext>if</mml:mtext></mml:mrow></mml:mtd><mml:mtd columnalign="left"><mml:mrow><mml:mn>1</mml:mn><mml:mo>≤</mml:mo><mml:mi>j</mml:mi><mml:mo>≤</mml:mo><mml:mi>J</mml:mi><mml:mo>,</mml:mo></mml:mrow></mml:mtd></mml:mtr><mml:mtr columnalign="left"><mml:mtd columnalign="left"><mml:mn>0</mml:mn></mml:mtd><mml:mtd columnalign="left"><mml:mrow><mml:mtext>if</mml:mtext></mml:mrow></mml:mtd><mml:mtd columnalign="left"><mml:mrow><mml:mi>J</mml:mi><mml:mo>&lt;</mml:mo><mml:mi>j</mml:mi><mml:mo>≤</mml:mo><mml:mi>G</mml:mi><mml:mo>.</mml:mo></mml:mrow></mml:mtd></mml:mtr></mml:mtable></mml:mrow></mml:mrow></mml:mrow></mml:semantics></mml:math></disp-formula> <disp-formula id="bmcM7"><label>(7)</label><italic>M </italic>= 100 to 1000</disp-formula> <disp-formula id="bmcM8"><label>(8)</label><italic>G </italic>= 5000</disp-formula> <disp-formula id="bmcM9"><label>(9)</label><italic>J </italic>= {500, 2500, 5000}</disp-formula> <disp-formula id="bmcM10"><label>(10)</label><italic>μ</italic><sub>0 </sub>= .1</disp-formula> <disp-formula id="bmcM11"><label>(11)</label><italic>P</italic>(<italic>ω</italic><sub>1</sub>) = .5</disp-formula> <disp-formula id="bmcM12"><label>(12)</label><italic>P</italic>(<italic>ω</italic><sub>2</sub>) = .5</disp-formula>
<supplementary-material content-type="local-data" id="S1"><caption><title>Additional file 1</title><p>Information on summing SNRs.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S2"><caption><title>Additional file 2</title><p>Scatterplot indicating the classification error relative to the number of datasets that is pooled, using a K Nearest Neighbor classifier (K-NN, K = 3). A) DLCV error. B) Error on a large independent validation set of 2000 samples. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S3"><caption><title>Additional file 3</title><p>Scatterplot indicating the classification error relative to the number of datasets that is pooled, using a Support Vector Machine classifier (SVM-RBF). A) DLCV error. B) Error on a large independent validation set of 2000 samples. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S4"><caption><title>Additional file 4</title><p>Scatterplot indicating the classification error relative to the number of samples that is pooled. A) DLCV error. B) Error on the Vijver <italic>et al. </italic>[##REF##12490681##3##] dataset. C) Number of genes selected by the DLCV protocol. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S5"><caption><title>Additional file 5</title><p>Scatterplot indicating the classification error relative to the number of datasets that is pooled, using a K Nearest Neighbor classifier (K-NN, K = 3). A) DLCV error. B) Error on the Vijver <italic>et al. </italic>[##REF##12490681##3##] dataset. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S6"><caption><title>Additional file 6</title><p>Scatterplot indicating the classification error relative to the number of datasets that is pooled, using a Support Vector Machine Classifier (Radial Basis Function used as kernel). A) DLCV error. B) Error on the Vijver <italic>et al. </italic>[##REF##12490681##3##] dataset. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S7"><caption><title>Additional file 7</title><p>Network indicating the synergy between six real datasets (ER positive samples only). Each node represents a dataset, and each edge the effect on the DLCV error when pooling them. Four different effects were considered, synergy (bright green) when the pooled error is lower than each of the separate errors. Marginal synergy (light blue) when the pooled error is lower than the weighted mean of the separate errors, conversely marginal anti-synergy (yellow) when it is higher. Lastly, true anti-synergy (orange) indicates a higher DLCV error for the pooled dataset.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S8"><caption><title>Additional file 8</title><p>Scatterplot indicating the classification error relative to the number of datasets that is pooled (ER positive samples only). A) DLCV error. B) Error on the Vijver <italic>et al. </italic>[##REF##12490681##3##] dataset. C) Number of genes selected by the DLCV protocol. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S9"><caption><title>Additional file 9</title><p>Heatmap of the Bonferroni corrected p-values of the enrichment between each signature and a collection of gene sets (ER postive samples only). Only categories with at least 1 significant association are shown.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S10"><caption><title>Additional file 10</title><p>Network indicating the synergy between six real datasets (ER negative samples only). Each node represents a dataset, and each edge the effect on the DLCV error when pooling them. Four different effects were considered, synergy (bright green) when the pooled error is lower than each of the separate errors. Marginal synergy (light blue) when the pooled error is lower than the weighted mean of the separate errors, conversely marginal anti-synergy (yellow) when it is higher. Lastly, true anti-synergy (orange) indicates a higher DLCV error for the pooled dataset.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S11"><caption><title>Additional file 11</title><p>Scatterplot indicating the classification error relative to the number of datasets that is pooled (ER negative samples only). A) DLCV error. B) Error on the Vijver <italic>et al. </italic>[##REF##12490681##3##] dataset. C) Number of genes selected by the DLCV protocol. The color corresponds to the number of datasets that was used. Labels indicate which combination of datasets was used.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S12"><caption><title>Additional file 12</title><p>Heatmap of the Bonferroni corrected p-values of the enrichment between each signature and a collection of gene sets (ER negative samples only). Only categories with at least 1 significant association are shown.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S13"><caption><title>Additional file 13</title><p>Centroids for the 127 gene classifier that was extracted from the six pooled datasets, including detailed info for the selected reporters.</p></caption></supplementary-material> <supplementary-material content-type="local-data" id="S14"><caption><title>Additional file 14</title><p>Indication of the distribution of various clinical parameters. In all cases the number of samples (#), and percentage of samples (%) is indicated, except for the tumor size (represented as mm).</p></caption></supplementary-material>
<table-wrap-foot><p>The Survival column indicates the type of survival data that was used; DMFS Distant Metastasis Free Survival; SOS breast cancer Specific Overall Survival. Lastly, for each dataset the total number of samples is indicated along with the number of poor/good samples per ER subgroup.</p></table-wrap-foot>
<graphic xlink:href="1471-2164-9-375-1"/> <graphic xlink:href="1471-2164-9-375-2"/> <graphic xlink:href="1471-2164-9-375-3"/> <graphic xlink:href="1471-2164-9-375-4"/> <graphic xlink:href="1471-2164-9-375-5"/> <graphic xlink:href="1471-2164-9-375-6"/> <graphic xlink:href="1471-2164-9-375-7"/> <graphic xlink:href="1471-2164-9-375-8"/> <graphic xlink:href="1471-2164-9-375-9"/> <graphic xlink:href="1471-2164-9-375-10"/>
<media xlink:href="1471-2164-9-375-S1.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S2.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S3.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S4.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S5.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S6.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S7.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S8.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S9.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S10.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S11.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S12.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S13.xls" mimetype="application" mime-subtype="vnd.ms-excel"><caption><p>Click here for file</p></caption></media> <media xlink:href="1471-2164-9-375-S14.pdf" mimetype="application" mime-subtype="pdf"><caption><p>Click here for file</p></caption></media>
[{"surname": ["Michiels", "Koscielny", "Hill"], "given-names": ["S", "S", "C"], "article-title": ["Prediction of Cancer Outcome With Microarrays: A Multiple Random Validation Strategy"], "source": ["The Lancet"], "year": ["2005"], "volume": ["365"], "fpage": ["488"], "lpage": ["92"], "pub-id": ["10.1016/S0140-6736(05)17866-0"]}, {"surname": ["Leek", "Storey"], "given-names": ["JT", "JD"], "article-title": ["Capturing heterogeneity in gene expression studies by surrogate variable analysis"], "source": ["PLoS Genet"], "year": ["2007"], "volume": ["3"], "fpage": ["e161"], "pub-id": ["10.1371/journal.pgen.0030161"]}, {"surname": ["Bolstad", "Collin", "Brettschneider"], "given-names": ["B", "F", "J"], "article-title": ["Quality assessment of affymetrix genechip data"], "source": ["Bioinformatics and Computational Biology Solutions Using R and Bioconductor"], "year": ["2005"], "publisher-name": ["Springer, NY"]}]
BMC Genomics. 2008 Aug 6; 9:375