Statistical significance for genomewide studies John D. Storey * and Robert Tibshirani * Department of Biostatistics , University of Washington , Seattle , WA 98195 ; and Departments of Health Research and Policy and Statistics , Stanford University , Stanford , CA 94305 Edited by Philip P. Green , University of Washington School of Medicine , Seattle , WA , and approved May 30 , 2003 -LRB- received for review January 28 , 2003 -RRB- With the increase in genomewide experiments and the sequencing of multiple genomes , the analysis of large data sets has become commonplace in biology . It is often the case that thousands of features in a genomewide data set are tested against some null hypothesis , where a number of features are expected to be significant . Here we propose an approach to measuring statistical significance in these genomewide studies based on the concept of the false discovery rate . This approach offers a sensible balance between the number of true and false positives that is automatically calibrated and easily interpreted . In doing so , a measure of statistical significance called the q value is associated with each tested feature . The q value is similar to the well known p value , except it is a measure of significance in terms of the false discovery rate rather than the false positive rate . Our approach avoids a flood of false positive results , while offering a more liberal criterion than what has been used in genome scans for linkage . false discovery rates genomics multiple hypothesis testing q values ome of the earliest genomewide studies involved testing for linkage at loci spanning a large portion of the genome . Because a separate statistical test is performed at each locus , traditional p-value cutoffs of 0.01 or 0.05 had to be made stricter to avoid an abundance of false positive results . The threshold for significance in linkage analysis is usually chosen so that the probability of any single false positive among all loci tested is 0.05 . This strict criterion is used mainly because one or very few loci are expected to show linkage in any given study -LRB- 1 , 2 -RRB- . Because of the recent surge in high-throughput technologies and genome projects , many more types of genomewide studies are now underway . The analyses of these data also involve performing statistical tests on thousands of features in a genome . As opposed to the linkage case , it is expected that many more than one or two of the tested features are statistically significant . Guarding against any single false positive occurring is often going to be much too strict and will lead to many missed findings . The goal is therefore to identify as many significant features in the genome as possible , while incurring a relatively low proportion of false positives . We are specifically concerned with situations in which a well defined statistical hypothesis test is performed on each of thousands of features represented in a genome . These `` features '' can be genes , all nucleotide words of a certain length , single-nucleotide polymorphism markers , etc. . Several motivating examples are given below . For each feature , a null hypothesis is tested against an alternative hypothesis . In this work , we say that a feature is truly null if the null hypothesis is true , and a feature is truly alternative if the alternative hypothesis is true . If a feature is called significant , then the null hypothesis is rejected in favor of the alternative hypothesis . The goal is to propose and estimate a measure of significance for each feature that meets the practical goals of the genomewide study and that is easily interpreted in terms of the simultaneous testing of thousands of features . We propose that the recently introduced q value -LRB- 3 , 4 -RRB- is a well suited measure of significance for this growing class of genomewide tests of significance . The q value is an extension of a quantity called the `` false discovery rate '' -LRB- FDR -RRB- -LRB- 5 -RRB- , which has received much recent attention in the statistics literature -LRB- 6 11 -RRB- . A FDR method has been used in detecting differential gene expression in DNA microarray experiments -LRB- 12 -RRB- , which can be shown to be equivalent 9440 9445 PNAS August 5 , 2003 vol . 100 no. 16 S to the method in ref . 5 under certain assumptions . Also , ideas similar to FDRs have appeared in the genetics literature -LRB- 1 , 13 -RRB- . Similarly to the p value , the q value gives each feature its own individual measure of significance . Whereas the p value is a measure of significance in terms of the false positive rate , the q value is a measure in terms of the FDR . The false positive rate and FDR are often mistakenly equated , but their difference is actually very important . Given a rule for calling features significant , the false positive rate is the rate that truly null features are called significant . The FDR is the rate that significant features are truly null . For example , a false positive rate of 5 % means that on average 5 % of the truly null features in the study will be called significant . A FDR of 5 % means that among all features called significant , 5 % of these are truly null on average . The q value provides a measure of each feature 's significance , automatically taking into account the fact that thousands are simultaneously being tested . Suppose that features with q values 5 % are called significant in some genomewide test of significance . This results in a FDR of 5 % among the significant features . A p-value threshold of 5 % yields a false positive rate of 5 % among all null features in the data set . In light of the definition of the false positive rate , a p-value cutoff says little about the content of the features actually called significant . The q values directly provide a meaningful measure among the features called significant . Because significant features will likely undergo some subsequent biological verification , a q-value threshold can be phrased in practical terms as the proportion of significant features that turn out to be false leads . Here we show that the FDR is a sensible measure of the balance between the number of true positives and false positives in many genomewide studies . We motivate our proposed approach in the context of several recent and prominent papers in which awkwardly chosen p-value cutoffs were used in an attempt to achieve at least qualitatively what the q value directly achieves . We also introduce a fully automated method for estimating q values , with an initial treatment of dependence issues between the features and guidelines as to when the estimates are accurate . The proposed methodology is applied to some gene expression data taken from cancer tumors -LRB- 14 -RRB- , supporting previously shown results and providing some additional information . Motivating Examples Consider the following four recent articles in which thousands of features from a genomewide data set were tested against a null hypothesis . In each case , p-value thresholds were used to decide which features to call significant , the ultimate goal being to identify many truly alternative features without including too many false positives . Example 1 : Detecting Differentially Expressed Genes . A common goal in DNA microarray experiments is to detect genes that show differential expression across two or more biological conditions -LRB- 15 -RRB- . In this scenario , the `` features '' are the genes , and they are tested against the null hypothesis that there is no differential This paper was submitted directly -LRB- Track II -RRB- to the PNAS office . Abbreviations : FDR , false discovery rate ; pFDR , positive FDR . To whom correspondence should be addressed . E-mail : jstorey@u.washington.edu . www.pnas.org cgi doi 10.1073 pnas .1530509100 gene expression . One of the goals of Hedenfalk et al. -LRB- 14 -RRB- was to find genes that are differentially expressed between BRCA1 - and BRCA2-mutation-positive tumors by obtaining several microarrays from each cell type . In their analysis they computed a modified F statistic and used it to assign a p value to each gene . A p-value cutoff of 0.001 was selected to find 51 genes of 3,226 that show differential gene expression . A rough calculation shows that about three false positives are expected with this cutoff . These authors later used a threshold of 0.0001 and concluded that 9 11 genes are differentially expressed . Example 2 : Identifying Exonic Splicing Enhancers . Exonic splice enhancers are short oligonucleotide sequences that enhance premRNA splicing when present in exons -LRB- 16 -RRB- . Fairbrother et al. -LRB- 17 -RRB- analyzed human genomic DNA to predict exonic splice enhancers based on the statistical analysis of exon intron and splice-site composition . They assessed the statistical significance of all 4,096 possible hexamers , the null hypothesis being a mathematical formulation of a hexamer not being an exonic splice enhancer . A statistic is formed based on the location of the hexamers in 4,817 human genes where the exon intron structure has been well characterized . The end product is a p value associated with each of the 4,096 hexamers . A p-value cutoff of 10 4 was used based on the rationale that , at most , 4,096 10 4 1 false positive is expected under this criterion . This cutoff yields 238 significant hexamers , a number of which were subsequently biologically verified . Example 3 : Genetic Dissection of Transcriptional Regulation . Global monitoring of gene expression and large-scale genotyping were recently used to study transcriptional regulation in yeast . Brem et al. -LRB- 18 -RRB- crossed two strains of yeast , where many genes appeared to be expressed differentially between these two strains . For 40 of the resulting haploid progeny , the expression levels of 6,215 genes were measured by using microarrays . Linkage was tested between 3,312 markers spanning the genome and each of these 6,215 `` quantitative traits . '' A statistically significant linkage between a gene 's expression level and a marker indicates that a regulator for that gene is located in the region of the marker . In analyzing these data , one can perform a statistical test for each gene-marker combination , resulting in millions of p values , or one can test each gene for showing linkage to at least one locus , resulting in 6,215 p values . Taking the latter approach and using a p-value cutoff of 8.5 10 3 , Brem et al. reported that 507 genes show linkage to at least one locus , where 53 are expected by chance . A cutoff of 1.6 10 4 yields 205 genes showing linkage to at least one locus , where 1 is expected by chance . The p values are calculated according to a permutation scheme to capture the dependence between adjacent markers -LRB- 19 -RRB- . The above-mentioned cutoffs correspond to respective thresholds of 5 10 5 and 2 10 6 when testing every gene marker combination . Several other p-value cutoffs with similar pieces of information are given throughout ref . 18 . Example 4 : Finding Binding Sites of Transcriptional Regulators . Transcriptional regulatory proteins bind to specific promoter sequences to participate in the regulation of gene expression . The availability of complete genome sequences and the development of a method for genomewide binding analysis has allowed the characterization of genomic sites bound by specific transcriptional regulators . Lee et al. -LRB- 20 -RRB- used genomewide location analysis to investigate how yeast transcriptional regulators bind to promoter sequences across the genome . Specifically , binding of 106 transcriptional factors was measured across the genome . At each genomic location , a p value was calculated under the null hypothesis that no binding occurs , resulting in the consideration of thousands of p values . Lee et al. `` generally describe results obtained at a p-value threshold of 0.001 because -LSB- their -RSB- analysis indicates that this threshold maximizes inclusion of legitimate regulator DNA interactions and minimizes Storey and Tibshirani Table 1 . Possible outcomes from thresholding m features for significance Called significant Null true Alternative true Total F T S Called not significant m0 m1 m F T S Total m0 m1 m Proposed Method and Results The dilemma of how to consider , say , m p values is seen more clearly by considering the various outcomes that occur when a significance threshold is applied to them . Table 1 lists these outcomes : specifically , F is the number of false positives , T is the number of true positives , and S is the total number of features called significant . Also , m0 is the number of truly null features in the study , and m1 m m0 is the number of truly alternative features . These quantities can be used to form an overall error measure for any given p-value cutoff . Regardless of whether the p-value threshold is fixed or data-dependent , the quantities F , T , and S are random variables . Therefore , it is common statistical practice to write the overall error measure in terms of an expected value , which we denote by E -LSB- -RSB- . If the false positive rate is the error measure used , then a simple p-value threshold is used . A p-value threshold of 0.05 , for example , guarantees only that the expected number of false positives is E -LSB- F -RSB- 0.05 m . This number is much too large for all of the examples we have considered , and the false positive rate is too liberal . The error measure that is typically controlled in genome scans for linkage is the familywise error rate , which can be written as Pr -LRB- F 1 -RRB- . -LSB- Note that we can guarantee that Pr -LRB- F 1 -RRB- by calling all features significant with p values m , which is the well known Bonferroni correction . -RSB- Controlling Pr -LRB- F 1 -RRB- is practical when very few features are expected to be truly alternative -LRB- e.g. , in the linkage case -RRB- , because any false positive can lead to a large waste of time . However , the familywise error rate is much too conservative for PNAS August 5 , 2003 vol . 100 no. 16 9441 GENETICS STATISTICS false positives . '' They estimate that among the 3,985 interactions found to be significant at this threshold , 6 10 % are false positives . Reasonable p-value thresholds were sought in each of the four examples . Three of them used four or more cutoffs in an attempt to circumvent the inherent difficulty in interpreting a p-value threshold in a genomewide study . The significance of the results is consequently obfuscated by the multiple cutoffs that are applied to the p values . Two pieces of information make such analyses more straightforward and universally interpretable . The first is an estimate of the overall proportion of features that are truly alternative , even if these can not be precisely identified . For example , what proportion of the 3,226 genes in example 1 are differentially expressed ? The second is a measure of significance that can be associated with each feature so that thresholding these numbers at a particular value has an easy interpretation . We provide both of these in our proposed approach . Note that , in example 1 , one could just as well work with the modified F statistic and threshold it directly . Directly thresholding the F statistic is equivalent to thresholding the p values described above . The proposed methodology described in terms of the original statistics can be intuitively pleasing for certain cases , proving that p values are not a necessary intermediate step . However , in other cases , such as examples 2 and 3 , the test statistics and null distributions are much more complicated , and p values provide a convenient numerical measure of the strength of evidence against the null for each feature . For this reason , we describe our proposal in terms in p values rather than test statistics . It is also preferable to present the q-value estimates in terms of p values to make the method widely applicable . However , working with the original test statistics and null distributions will lead to the same q-value estimates -LRB- 3 -RRB- . many of the genomewide studies currently being performed , including the four examples we considered where many features are expected to be truly alternative . It is therefore useful to find an error measure in between these , specifically , one that provides a sensible balance between the number of false positive features , F , and the number of true positive features , T . This balance can be achieved efficiently by considering the ratio no . false positive features no . significant features F F T F , S which can be stated in words as the proportion of false positive features among all of those called significant . We are particularly interested in the FDR , which is defined to be the expected value of this quantity : FDR E F F T E F . S Fig. 1 . A density histogram of the 3,170 p values from the Hedenfalk et al. -LRB- 14 -RRB- data . The dashed line is the density histogram we would expect if all genes were null -LRB- not differentially expressed -RRB- . The dotted line is at the height of our estimate of the proportion of null p values . To be completely rigorous , there is the possibility that S 0 , in which case F S is undefined , so some adjustment has to be made to this definition -LRB- see Remark A in Appendix -RRB- . The FDR can also be written in terms of the well known specificity , -LRB- m0 F -RRB- m0 , and sensitivity , T m1 : FDR E m0 1 m0 1 specificity . specificity m1 sensitivity F t # null pi St # pi t ; i t ; i 1 , ... , m and 1 , ... , m . Clearly , the FDR is a useful measure of the overall accuracy of a set of significant features for the examples we described and many other genomewide studies . But one would also like a measure of significance that can be attached to each individual feature . The q value is a measure designed to reflect this level of attachment . Suppose that we list the features in order of their evidence against the null hypothesis . It is practical to arrange the features in this way because calling one feature significant means that any other feature with more evidence against the null should also be called significant . Hence , we list the features from smallest to largest p value . If a threshold value is chosen , we call all features significant up through that threshold . The q value for a particular feature is the expected proportion of false positives incurred when calling that feature significant . Therefore , calculating the q values for each feature and thresholding them at q-value level produces a set of significant features so that a proportion of is expected to be false positives . Typically , the p value is described as the probability of a null feature being as or more extreme than the observed one . `` As or more extreme '' in this setup means that it would appear higher on the list . The q value of a particular feature can be described as the expected proportion of false positives among all features as or more extreme than the observed one . The q value has a special probabilistic relationship to the p value -LRB- yielding the origin of its name -RRB- that is briefly explained in Remark A in Appendix . As a concrete example , we considered the data from ref . 14 to identify genes that are differentially expressed between BRCA1and BRCA2-mutation-positive tumors . Using a two-sample t statistic , we calculated a p value for each of 3,170 genes under the null hypothesis of no differential gene expression . See Remark C in Appendix for specific details . Fig. 1 shows a density histogram of the 3,170 p values . The dashed line is the density we would expect if all genes were null -LRB- not differentially expressed -RRB- , so it can be seen that many genes are differentially expressed . Given the definition of the q value , it makes sense to begin by estimating the FDR when calling all features significant whose p value is less than or equal to some threshold t , where 0 t 1 . Denote the m p values by p1 , p2 , ... , pm , and let 9442 www.pnas.org cgi doi 10.1073 pnas .1530509100 We then want to estimate FDR t E Ft . St Because we are considering many features -LRB- i.e. , m is very large -RRB- , it can be shown that FDR t E Ft St EFt . ESt -LSB- 1 -RSB- A simple estimate of E -LSB- S -LRB- t -RRB- -RSB- is the observed S -LRB- t -RRB- ; that is , the number of observed p values t . In estimating E -LSB- F -LRB- t -RRB- -RSB- , recall that p values corresponding to truly null hypotheses should be uniformly distributed . -LSB- If the null p values are not uniformly distributed , then one wants to err in the direction of overestimating p values -LRB- i.e , underestimating significance -RRB- . Correctly calculated p values are an important assumption underlying our methodology . See also Remark D in Appendix . -RSB- Thus , the probability a null p value is t is simply t , and it follows from Table 1 that E -LSB- F -LRB- t -RRB- -RSB- m0 t. Because the total number of truly null features m0 is unknown it has to be estimated . Equivalently , one can estimate the -LRB- more interpretable -RRB- proportion of features that are truly null , which we denote by 0 m0 m . It is difficult to estimate 0 without specifying the distribution of the truly alternative p values . However , exploiting the fact that null p values are uniformly distributed , a reasonable estimate can be formed . From Fig. 1 we can see that the histogram density of p values beyond 0.5 looks fairly flat , which indicates that there are mostly null p values in this region . The height of this flat portion actually gives a conservative estimate of the overall proportion of null p values . This can be quantified with ^ 0 # pi ; i m1 1 , ... , m , 0.5 , we estimate which involves the tuning parameter . Setting that 67 % of the genes in the data from ref . 14 are not differentially expressed . Note that through significance tests , prediction models , and various other techniques , it has been qualitatively argued that Storey and Tibshirani BRCA1 - and BRCA2-mutation-positive tumors can be distinguished by their genetic profiles -LRB- 14 -RRB- . Our estimate of 67 % provides a direct measurement of this ; we estimate that at least 33 % of the examined genes are differentially expressed between these two tumor types . Using traditional p-value cutoffs , Hedenfalk et al. -LRB- 14 -RRB- were comfortable only with concluding that 9 11 genes are differentially expressed of 3,000 . The rationale behind the estimate of 0 is that p values of truly alternative features will tend to be close to zero , whereas p values of null features will be uniformly distributed among -LSB- 0 , 1 -RSB- . `` Most '' of the p values we observe near 1 will be null then . If we were able to count only null p values , then # -LCB- null pi -RCB- m -LRB- 1 -RRB- would be an unbiased estimate of 0 . The inclusion of a few alternative p values only makes this estimate conservative . If we take 0 , then ^ 0 -LRB- -RRB- 1 , which is usually going to be much too conservative in genomewide data sets , where a sizable proportion of features are expected to be truly alternative . However , as we set closer to 1 , the variance of ^ 0 -LRB- -RRB- increases , making the estimated q values more unreliable . By examining the data in Fig. 1 , a common sense choice for was 0.5 . In general , it is useful to automate this choice . We introduce a fully automated method in Remark B in Appendix for estimating 0 that borrows strength across a range of ^ 0 -LRB- -RRB- . This automated method also happens to result in ^ 0 0.67 . By plugging these quantities into the right side of Eq . 1 , FDR -LRB- t -RRB- is estimated by ^ t FDR ^ 0m t St ^ 0m t . # pi t Fig. 2 . Results from the Hedenfalk et al. -LRB- 14 -RRB- data . -LRB- a -RRB- The q values of the genes versus their respective t statistics . -LRB- b -RRB- The q values versus their respective p values . -LRB- c -RRB- The number of genes occurring on the list up through each q value versus the respective q value . -LRB- d -RRB- The expected number of false positive genes versus the total number of significant genes given by the q values . The more mathematical definition of the q value is the minimum FDR that can be attained when calling that feature significant -LRB- see Remark A in Appendix -RRB- . Thus , the q value of feature i is mint pi FDR -LRB- t -RRB- , where we have simply considered all thresholds t pi . We ^ can estimate the q value of feature i by simply plugging FDR -LRB- t -RRB- into the definition above : q pi ^ ^ min FDR t . t pi Note that this guarantees that the estimated q values are increasing in the same order as the p values . This method is presented in an easily implemented and fully automated algorithm in Remark B in Appendix . We mention two mathematical results concerning the accuracy of the estimated q values that hold for large m under what we call `` weak dependence '' of the p values -LRB- or features -RRB- . Weak dependence can loosely be described as any form of dependence whose effect becomes negligible as the number of features increases to infinity . -LRB- See Remark D in Appendix and ref . 10 . -RRB- The first result is that if we call all features significant with q values , then for large m the FDR will be . The second result is that the estimated q values are simultaneously conservative for the true q values . This means that the estimated q value of each feature is greater than or equal to its true q value , across all features at once . Under this result , one can consider each feature 's significance simultaneously without worrying about inducing bias . In a sense , the second result implies that one can consider all cutoffs simultaneously , which is a much stronger generalization of the first result . These conservative properties are desirable because one does not want to underestimate the true q values or the true proportion of false positives . We hypothesize that the most likely form of dependence between features in a genomewide data set will meet the weak dependence requirement , although this has to be considered for each application . Specifically for DNA microarray data , we argue that because genes behave dependently in small groups -LRB- i.e. , pathways -RRB- , with each group essentially being independent of the others , then the dependence can be modeled in blocks in such a way to satisfy the mathematical conditions . More specific details of these mathematical results can be found in Remark D in Appendix . Storey and Tibshirani Given this potentially valuable theoretical justification for considering all q values simultaneously , even in the presence of weak dependence , it is possible to use several plots to calibrate the q-value cutoff one would want to apply in a study . -LRB- On the other hand , a single cutoff is not always necessary ; each feature 's estimated q value could simply be reported . -RRB- Fig. 2a shows a plot of the q values versus their t statistics from the data in ref . 14 . Fig. 2b is a plot of the q values versus their p values . One can see the expected proportion of false positives for different p-value cutoffs from this plot . Fig. 2c shows the number of significant genes for each q value . Notice that for estimated q values slightly greater than 0.02 , a sharp increase occurs in the number of significant genes over a small increase in q value . This allows one to easily see that a slightly larger q-value cutoff results in many more significant genes . Finally , Fig. 2d shows the expected number of false positives as a function of the number of genes called significant . In general , these last three plots can be used concurrently to give the researcher a comprehensive view of what features to examine further . In our analysis , thresholding genes with q values 0.05 yields 160 genes significant for differential expression . This means that 8 of the 160 genes called significant are expected to be false positives . It has previously been noticed that a large block of genes are overexpressed in BRCA1-mutation-positive tumors , in particular , genes involved in DNA repair and apoptosis -LRB- 14 -RRB- . We find that 117 of the 160 called significant at q-value level 0.05 are overexpressed in BRCA1-mutation-positive tumors , quantitatively supporting their claim . The 0.05 q-value cutoff is arbitrary , and we do not recommend that this value necessarily be used . Considering particular genes allows us to examine their individual q values . For example , the MSH2 gene -LRB- clone 32790 -RRB- is the eighth most significant gene for differential expression with a q value of 0.013 and a p value of 5.05 10 5 . This gene is overexpressed in the BRCA1-mutation-positive tumors , indicating increased levels of DNA repair -LRB- 21 -RRB- . MSH2 's p value of 5.05 10 5 says that the probability a null -LRB- nondifferentially expressed -RRB- gene would be as or more extreme than MSH2 is 5.05 10 5 . But MSH2 's statistic could also be unlikely for a differentially expressed gene . The q value allows a quantification of this ; the estimated q value for MSH2 is 0.013 , meaning that 0.013 of the genes that are as or more extreme than MSH2 are false positives . The PDCD5 gene -LRB- clone 502369 -RRB- is the 47th most significant gene , with a q value of 0.022 and p value of 4.79 10 4 . This gene , associated with inducing apoptosis -LRB- 15 -RRB- , is also overexpressed in BRCA1-mutation-positive tumors . The PNAS August 5 , 2003 vol . 100 no. 16 9443 GENETICS STATISTICS CTGF gene -LRB- clone 38393 -RRB- is the 159th most significant gene for differential expression -LRB- q value 0.049 ; p value 0.0036 -RRB- and is overexpressed in BRCA2-mutation-positive tumors . Activity of this gene is associated with suppressing apoptosis -LRB- 23 -RRB- , which further supports earlier claims -LRB- 14 -RRB- . Therefore , our results support the previous observation that many genes are overexpressed in BRCA1mutation-positive tumors , particularly genes involved in DNA repair and apoptosis . A full list of genes with their q values , p values , and fold change is available at http : genomine.org qvalue results.html . A common mistake is to state that the p value is the probability a feature is a false positive . We stress that the q value is also not the probability that the feature is a false positive . In the example presented above MSH2 has a q value equal to 0.013 . This value does not imply that MSH2 is a false positive with probability 0.013 . Rather , 0.013 is the expected proportion of false positives incurred if we call MSH2 significant . Because the q-value measure includes genes that are possibly much more significant than MSH2 , the probability that MSH2 is itself a false positive may be substantially higher . In terms of the FDR approach , this probability can also be thought of as a `` local FDR '' -LRB- 3 , 8 , 24 , 25 -RRB- . Statistical significance involves deciding between null and alternative hypotheses . When assigning multiple measures of statistical significance , it is necessary to account for the fact that decisions are made for m features simultaneously . The q value accomplishes this by conditioning based on the fact that every feature as or more extreme will also be called significant : the probability a feature is a false positive does not . However , the latter quantity clearly provides very useful information , and ideally one would have both estimates available for the analysis of a genomewide study . Discussion We have proposed the q value as an FDR-based measure of significance for genomewide studies . The methodology we have proposed is the only methodology theoretically shown to be conservative -LRB- over all q values -RRB- in situations plausibly encountered in genomics . -LRB- See Remark D in Appendix and ref . 10 . -RRB- The proposed methodology is easy to implement and interpret , and it is fully automated . The original FDR methodology -LRB- 5 -RRB- is too conservative for genomics applications because it assumes 0 1 . For example , controlling the FDR at 0.03 , 0.05 , or 0.07 in the expression data -LRB- 14 -RRB- finds 80 , 160 , or 231 significant genes , respectively , when our proposed method is used . The methodology in ref . 5 finds only 21 , 88 , or 153 , respectively , indicating that this earlier method 's estimates are too conservative and result in a substantial loss of power . The approach in ref . 5 also forces one to choose a single acceptable FDR level before any data are seen , which is often going to be impractical and too restrictive . The q value of a particular feature in a genomewide data set is the expected proportion of false positives incurred when calling that feature significant . One may use the q values as an exploratory guide for which features to investigate further . One may also take . all features with q values some threshold to attain a FDR Most importantly , a systematic use of q values in genomewide tests of significance will yield a clear balance of false positives to true positive results and give a standard measure of significance that can be universally interpreted . The methodology we presented also provides an estimated ^ 0 of the proportion of features following the null hypothesis . The quantity ^ 1 1 ^ 0 estimates a lower bound on the proportion of truly alternative features . For example , among the 3,170 genes we examined from ref . 14 , we found that at least 33 % are differentially expressed between BRCA1 - and BRCA2mutation-positive tumors . Similar estimates from the other examples we considered would be interesting to compute . The software QVALUE can be downloaded at http : genomine . org qvalue . This program takes a list of p values and computes their estimated q values and ^ 0 . A version of Fig. 2 is also generated . 9444 www.pnas.org cgi doi 10.1073 pnas .1530509100 Fig. 3 . The ^ 0 -LRB- -RRB- versus for the data of Hedenfalk et al. -LRB- 14 -RRB- . The solid line 1 -RRB- . is a natural cubic spline fit to these points to estimate ^ 0 -LRB- Appendix Remark A : FDR , Positive FDR -LRB- pFDR -RRB- , and the q Value . In this article , we have used FDR and FDR E -LSB- F S -RSB- somewhat loosely . It will almost always be the case that S 0 with positive probability , which implies that E -LSB- F S -RSB- is undefined . The quantity E -LSB- F S S 0 -RSB- Pr -LRB- S 0 -RRB- was proposed as a solution to this problem -LRB- 5 -RRB- , which is the result of setting F S 0 whenever S 0 in the original E -LSB- F S -RSB- . This quantity is technically called the FDR in the statistics literature . In our case we want to place a measure of significance on each feature , which is done under the assumption that the feature is called significant . Thus , the inclusion of Pr -LRB- S 0 -RRB- is somewhat awkward . An alternative quantity , called the pFDR , was recently proposed -LRB- 23 -RRB- , which is simply defined as pFDR E -LSB- F S S 0 -RSB- . The q value is most technically defined as the minimum pFDR at which the feature can be called significant -LRB- 24 -RRB- . Because m is large in genomewide studies , we have that Pr -LRB- S 0 -RRB- 1 and FDR pFDR E -LSB- F -RSB- E -LSB- S -RSB- , so the distinction is not crucial here . Also , the estimate we use is easily motivated for either quantity -LRB- 4 , 10 -RRB- . Suppose that each feature 's statistic probabilistically follows a random mixture of a null distribution and an alternative distribution . Then under a fixed significance rule , the pFDR can be written as Pr -LRB- feature i is truly null feature i is significant -RRB- , for any i 1 , ... , m -LRB- 3 -RRB- . Similarly , the false positive rate can be written as Pr -LRB- feature i is significant feature i is truly null -RRB- , for any i 1 , 2 , ... , m. Notice the similarity between the pFDR and false positive rate : the arguments have simply been swapped in the conditional probabilities . This connection is the motivation for calling our proposed quantity q value . Indeed , the p value of a feature is technically defined to be the minimum possible false positive rate when calling that feature significant -LRB- 26 -RRB- . Likewise , the q value is based on the minimum possible pFDR . tradeoff between bias and variance in choosing the to use in ^ 0 -LRB- -RRB- . For well formed p values , it should be the case that the bias of ^ 0 -LRB- -RRB- decreases with increasing , the bias being the smallest when 3 1 -LRB- 4 -RRB- . Therefore , the method we use here is to estimate ^ 0 -LRB- 1 -RRB- . In doing so , we will borrow strength lim 31 ^ 0 -LRB- -RRB- across the ^ 0 -LRB- -RRB- over a range of , giving an implicit balance between bias and variance . 0 , Consider Fig. 3 , where we have plotted ^ 0 -LRB- -RRB- versus for 0.01 , 0.02 , ... , 0.95 . By fitting a natural cubic spline to these data -LRB- solid line -RRB- , we have estimated the overall trend of ^ 0 -LRB- -RRB- as increases . We purposely set the degrees of freedom of the natural cubic spline to 3 ; this means we limit its curvature to be like a Storey and Tibshirani Remark B : General Algorithm for Estimating q Values . There is a quadratic function , which is suitable for our purposes . It can be seen from Fig. 3 that the natural cubic spline fits the points quite well . 1 is our final estimate of The natural cubic spline evaluated at 0 . For a variety of simulations and forms of dependence -LRB- data not shown -RRB- , this method performed well , often eliminating all bias in ^ 0 . The following is the general algorithm for estimating q values from a list of p values . 1 . Let p -LRB- 1 -RRB- p -LRB- 2 -RRB- ... p -LRB- m -RRB- be the ordered p values . This also denotes the ordering of the features in terms of their evidence against the null hypothesis . 2 . For a range of , say 0 , 0.01 , 0.02 , ... , 0.95 , calculate ^ 0 # pj m1 . for i 1 , 2 , ... , 3,170 . We next calculated null versions of t1 , t2 , ... , t3170 when there is no differential gene expression . Because it is not clearly valid to assume that the ti follow a t distribution , we calculate these by a permutation method . Consider all possible ways to assign n 15 arrays to n1 7 arrays from BRCA1 and n2 8 arrays from BRCA2 . Under the assumption that there is no differential gene expression , the t statistic should have the same distribution regardless of how we make these assignments . Specifically , the labels on the arrays are randomly scrambled , and the t statistics are recomputed . Therefore , for B 100 permutations of the array labels we get a set of null statistics t0b , ... , t0b , b 1 , ... B . The p value for gene i , i 1 , 1 3170 2 , ... , 3,170 was calculated by B 3 . Let ^ be the natural cubic spline with 3 df of ^ 0 -LRB- -RRB- on . f 4 . Set the estimate of 0 to be ^ 0 5 . Calculate q pm ^ 6 . For i m 1 , m t p pi b 1 # j : t0b j ti , j 1 , ... , 3170 . 3170 B ^ 1 . f ^ 0m t # pj t min m ^ 0 p m . We estimated the q values for differential gene expression between the BRCA1 and BRCA2 tumors by using the algorithm presented above . All results , including the computer code used to analyze the data , can be found at http : genomine.org qvalue results.html . under `` weak dependence '' of the p values -LRB- or features in the genome -RRB- . These mathematical results indicate that our method yields conservative q-value estimates . The conservative property is desirable because one does not want to underestimate the true q values . -LRB- For the same reason one would not want to underestimate a p value . -RRB- Suppose that with probability 1 , we have S -LRB- t -RRB- m 3 G -LRB- t -RRB- and F -LRB- t -RRB- m0 3 G0 -LRB- t -RRB- for each t -LSB- 0 , 1 -RSB- as m 3 , where G and G0 are continuous functions . In words , this says that the empirical distribution functions of the observed p values and null p values converge pointwise to some continuous functions . Weak dependence is defined as dependence that allows this pointwise convergence . -LRB- As a rule of thumb , the more local the dependence is , the more likely it is to meet the weak dependence criterion . -RRB- Also suppose that G0 -LRB- t -RRB- t -LRB- i.e. , uniform distribution or more conservative -RRB- , and that m0 m converges . If we constrain ^ 0 min R ^ 0 -LRB- -RRB- -LRB- which should e usually be the case -RRB- , then it can be shown that for any 0 , m3 2 , ... , 1 , calculate ^ 0m p i min , q pi ^ i 1 Remark D : Theoretical Properties . Some mathematical results hold q pi ^ ^ 0m t min t t p i # pj . 7 . The estimated q value for the ith most significant feature is q ^ -LRB- p -LRB- i -RRB- -RRB- . Remark C : Analysis of the Hedenfalk et al. . Data . The data from ref . 14 can be obtained at http : research.nhgri.nih.gov microarray NEJMSupplement . The data consist of 3,226 genes on n1 7 BRCA1 arrays and n2 8 BRCA2 arrays , along with some arrays from sporadic breast cancer , which we did not use . If any gene had one or more measurement exceeding 20 , then this gene was eliminated . A value of 20 is several interquartile ranges away from the interquartile range of all of the data and did not seem trustworthy for this example . This left m 3,170 genes . We tested each gene for differential expression between these two tumor types by using a two-sample t statistic . Let the log2 expression value from the jth array and the ith gene be denoted by 2 xij . Then xi2 1 n2 j BRCA2 xij and si2 1 -LRB- n2 1 -RRB- j BRCA2 -LRB- xij xi2 -RRB- 2 are the sample mean and variance for gene i among the arrays 2 taken from BRCA2 tumors . We can similarly define xi1 and si1 to be the sample mean and variance for the ith gene among the BRCA1 tumor arrays . The two-sample t statistic for the ith gene , allowing for the possibility that the tumors have different variances , is then lim min q pi ^ pi q value pi 0 . which means that the estimated q values are simultaneously conservative for the true q values , even when taking the worst-case scenario over -LSB- , 1 -RSB- for arbitrarily small . Also , we can conclude that lim , GENETICS ti x i2 s2 i1 n1 x i1 s2 i2 n2 m3 # false positive q pi ^ # q pi ^ which means that if we call all genes with q values , then in the long run the FDR will be . The proofs of these claims follow from minor modifications to some of the main results in ref . 10 . 15 . 16 . 17 . 18 . 19 . 20 . 21 . 22 . 23 . 24 . 25 . 26 . Slonim , D. K. -LRB- 2002 -RRB- Nat . Genet . 32 , Suppl. , 502 508 . Blencowe , B. J. -LRB- 2000 -RRB- Trends Biochem . Sci . 25 , 106 110 . Fairbrother , W. G. , Yeh , R. F. , Sharp , P. A. & Burge , C. B. -LRB- 2002 -RRB- Science 297 , 1007 1013 . Brem , R. B. , Yvert , G. , Clinton , R. & Kruglyak , L. -LRB- 2002 -RRB- Science 296 , 752 755 . Churchill , G. A. & Doerge , R. W. -LRB- 1994 -RRB- Genetics 138 , 963 971 . Lee , T. I. , Rinaldi , N. J. , Robert , F. , Odom , D. T. , Bar-Joseph , Z. , Gerber , G. K. , Hannett , N. M. , Harbison , C. T. , Thompson , C. M. , Simon , I. , et al. -LRB- 2002 -RRB- Science 298 , 799 804 . Kolodner , R. -LRB- 1996 -RRB- Genes Dev . 10 , 1433 1442 . Liu , H. T. , Wang , Y. G. , Zhang , Y. M. , Song , Q. S. , Di , C. H. , Chen , G. H. , Tang , J. & Ma , D. L. -LRB- 1999 -RRB- Biochem . Biophys . Res . Commun . 254 , 203 210 . Hishikawa , K. , Oemar , B. S. , Tanner , F. C. , Nakaki , T. , Luscher , T. F. & Fujii , T. -LRB- 1999 -RRB- J. Biol . Chem . 274 , 37461 37466 . Efron , B. , Storey , J. D. & Tibshirani , R. -LRB- 2001 -RRB- Technical Report 2001-217 -LRB- Stanford Univ. , Palo Alto , CA -RRB- . Efron , B. & Tibshirani , R. -LRB- 2002 -RRB- Genet . Epidemiol . 23 , 70 86 . Lehmann , E. L. -LRB- 1986 -RRB- Testing Statistical Hypotheses -LRB- Springer , New York -RRB- , 2nd Ed . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 10 . 11 . Morton , N. E. -LRB- 1955 -RRB- Am . J. Hum . Gen. 7 , 277 318 . Lander , E. S. & Kruglyak , L. -LRB- 1995 -RRB- Nat . Genet . 11 , 241 247 . Storey , J. D. -LRB- 2003 -RRB- Ann . Stat. , in press . Storey , J. D. -LRB- 2002 -RRB- J. R. Stat . Soc . B 64 , 479 498 . Benjamini , Y. & Hochberg , Y. -LRB- 1995 -RRB- J. R. Stat . Soc . B 85 , 289 300 . Yekutieli , D. & Benjamini , Y. -LRB- 1999 -RRB- J. Stat . Plan . Inf . 82 , 171 196 . Benjamini , Y. & Hochberg , Y. -LRB- 2000 -RRB- J. Ed . Behav . Stat . 25 , 60 83 . Efron , B. , Tibshirani , R. , Storey , J. D. & Tusher , V. -LRB- 2001 -RRB- J. Am . Stat . Assoc. 96 , 1151 1160 . Genovese , C. & Wasserman , L. -LRB- 2002 -RRB- J. R. Stat . Soc . B 64 , 499 517 . Storey , J. D. , Taylor , J. E. & Siegmund , D. -LRB- 2003 -RRB- J. R. Stat . Soc . B , in press . Tzeng , J. Y. , Byerley , W. , Devlin , B. , Roeder , K. & Wasserman , L. -LRB- 2003 -RRB- J. Am . Stat . Assoc. 98 , 236 246 . 12 . Tusher , V. , Tibshirani , R. & Chu , C. -LRB- 2001 -RRB- Proc . Natl. Acad . Sci . USA 98 , 5116 5121 . 13 . Tatusov , R. L. , Altschul , S. F. & Koonin , E. V. -LRB- 1994 -RRB- Proc . Natl. Acad . Sci . USA 91 , 12091 12095 . 14 . Hedenfalk , I. , Duggan , D. , Chen , Y. D. , Radmacher , M. , Bittner , M. , Simon , R. , Meltzer , P. , Gusterson , B. , Esteller , M. , Kallioniemi , O. P. , et al. -LRB- 2001 -RRB- N. Engl . J. Med . 344 , 539 548 . Storey and Tibshirani PNAS August 5 , 2003 vol . 100 no. 16 9445 STATISTICS

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B

microarray expression genetic fdr statistics gene