the application of data - mining algorithms to these large genomic datasets provides great potential for uncovering novel biology , but currently this potential is not often realized because collecting , properly processing and analyzing these data requires substantial computational resources and sophisticated programming knowledge . on the other hand ,
setting up analyses to address important biological questions and testing novel predictions resulting from such analyses requires detailed experimental knowledge .
although there are several successful applications of sophisticated computing approaches to diverse functional genomics data collections ( 15 ) , including some that share results through a web site ( 69 ) , currently there is not an easy way for a researcher to set up new analyses and ask specific biological questions by focusing these analyses on a sub - process or tissue of interest .
this greatly constrains the utility of the novel predictions , because direct experimental validation for some processes or tissues may be impractical .
pilgrm ( the platform for interactive learning by genomics results mining ) addresses this limitation by allowing its users to generate specific biological hypotheses by directing the supervised analyses of global microarray expression collections simply by defining their own gold standards ( lists of genes relevant to a process , disease or tissue ) .
such an approach puts sophisticated computational tools in the hands of biologists , thereby combining their biological insight with a powerful computational strategy .
this flexibility lets users address questions as diverse as their research programs while targeting predictions to experimentally testable pathways , tissues or phenotypes .
efforts to predict protein function , expression or localization from high - throughput data compendia generally make computational predictions based on annotations from expert - curated literature - derived databases .
these databases also do not represent unpublished experimental results that may be informative for future experiments . by encouraging and enabling users to define their own standards
indeed as the successful prior applications of data mining strategies to these compendia have shown , these databases have great value .
this is why pilgrm contains extensive collections of data and database - derived gold standards ( detailed in table 1 ) for homo sapiens and the model organisms mus musculus , rattus norvegicus , caenorhabditis elegans , arabidopsis thaliana and saccharomyces cerevisiae .
we include the gene ontology , which has annotations for a protein s biological process involvement , localization and biochemical function ( 10,11 ) , the plant ontology , which has annotations for a protein s role in plant development and anatomy ( 12 ) , the saccharomyces genome database phenotype annotations , which specify phenotypes observed when genes are knocked out ( 13 ) and the human protein reference database s tissue annotations , which provide literature - derived annotations of tissue specific expression , localization and function for human proteins ( 14 ) .
these database annotations provide a convenient starting point for user - defined standards and analyses . 

table 1.pilgrm contains large data compendia and standards derived from literature - curated databases for each the organisms that it coversexperimentsarraysgenesstandardsunique publicationshuman239277  47321  702748432  567mouse201231  37424  555686414  248yeast11718016077423110  134arabidopsis408546522  12139296836rat44010  37621  416524214  395worm5396317  02717822489the unique publications column shows how many distinct publications are represented in the gold standards pre - loaded in pilgrm for each organism .
pilgrm contains large data compendia and standards derived from literature - curated databases for each the organisms that it covers the unique publications column shows how many distinct publications are represented in the gold standards pre - loaded in pilgrm for each organism .
for example , a researcher studying breast cancer progression may be interested in identifying novel candidate genes involved in breast cancer progression while avoiding genes that appear relevant simply because they are expressed in mammary epithelium ( i.e. genes discoverable by a simple correlation analysis ) .
this researcher can take advantage of both custom standards and the included database annotations in pilgrm .
setting up such an analysis without pilgrm would require that he download the full collection of over 70  000 gene expression experiments for human , develop appropriate data processing , normalization and integration methods , and set up a machine - learning framework for the analysis .
he would then have to download the hprd database to identify genes known to be expressed in the mammary epithelium , in addition to creating his custom standard of genes involved in breast cancer progression .
in contrast , this analysis takes minutes in pilgrm : figure 1 shows the steps that this user performs during the preparation and interpretation of this analysis .
first , the researcher develops a gold standard for genes involved in breast cancer through his own expertise and a literature search ( figure 1a ) .
the pilgrm server allows each link between a gene and a gold standard to be associated with pubmed identifiers and these publication - annotated links are included in a downloadable document ( pdf format ) describing each analysis that is made available to the user ( such a document can be used for additional record keeping by the users , to inform a materials and methods section , or directly as supplementary data in publications resulting from this analysis ) .
second , he creates an analysis and pairs this breast cancer standard with the hprd mammary epithelium standard included in pilgrm ( figure 1b ) . as a final step ,
the researcher runs this analysis and both metrics for the machine - learning results and novel predictions are returned by pilgrm . 

figure 1.the flow of a pilgrm analysis that uses one custom standard and a pre - loaded standard to discover genes related to breast cancer progression while excluding general mammary epithelium genes .
( a ) the process of creating a standard and adding genes ( here shown with optional pubmed ids ) to it .
the breast cancer standard from ( a ) is combined with the hprd mammary epithelium standard that is pre - loaded into pilgrm .
the breast cancer standard is a positive and the mammary epithelium is a negative ( here both are shown together for clarity ) .
the analysis is run and standards quantitative performance metrics and novel predictions are provided to the user .
the flow of a pilgrm analysis that uses one custom standard and a pre - loaded standard to discover genes related to breast cancer progression while excluding general mammary epithelium genes .
( a ) the process of creating a standard and adding genes ( here shown with optional pubmed ids ) to it .
the breast cancer standard from ( a ) is combined with the hprd mammary epithelium standard that is pre - loaded into pilgrm .
the breast cancer standard is a positive and the mammary epithelium is a negative ( here both are shown together for clarity ) .
the analysis is run and standards quantitative performance metrics and novel predictions are provided to the user . by combining custom standards with appropriate literature - curated databases and sophisticated machine learning of the support vector machine ( svm ) classifier implemented in pilgrm , this researcher discovers genes relevant to his research and saves time without compromising the flexibility or quality of his data - driven predictions .
these relevant genes behave similarly ( e.g. through co - expression ) to the genes defined as interesting by the user ( the positive standard ) in informative experimental conditions .
the machine - learning approach automatically identifies the conditions that best differentiate positive standard genes from those in the negative standard ( genes with properties that the user wishes to avoid in new predictions ) .
pilgrm provides both novel predictions and high - quality interactive visualizations of analysis results for the researcher to explore .
pilgrm s main features are as follows . 
 a flexible interface that encourages user - defined data - driven analyses that answer diverse questions of biological interest including those outside the scope of common databases.regularly updated compendia of uniformly processed genomic data for human and common model organisms.regularly updated gold standards for tissue , function and development from common sources ( go , po , hprd , etc . ) that make setting up analyses quick and easy.user-set levels of access control ( public , hidden , private ) for standards and analyses , allowing users to include unpublished results in pilgrm . a flexible interface that encourages user - defined data - driven analyses that answer diverse questions of biological interest including those outside the scope of common databases .
regularly updated gold standards for tissue , function and development from common sources ( go , po , hprd , etc . ) that make setting up analyses quick and easy .
user - set levels of access control ( public , hidden , private ) for standards and analyses , allowing users to include unpublished results in pilgrm .
the user translates this question into appropriate gold standards , thereby defining the corresponding machine - learning problem .
gold standards are structured as positives ( which represent genes like those that the user is seeking ) and negatives ( which represent genes with properties the user wants to exclude ) and can be drawn from databases or developed by the user .
pilgrm then classifies all other genes in the organism of interest with a machine - learning algorithm that employs the user - provided positive and negative standards , thereby generating novel predictions .
this process is summarized in figure 2 and discussed in detail in supplementary data s1 . 

we pre - process separate datasets into a gene - expression compendium for each organism .
( b ) the user provides positive and negative standards ( either input by the user or from common databases ) and the data are labeled with these standards .
the svm algorithm identifies the maximum - margin hyperplane ( here a dotted line for the two - arrays in this example , but in practice this is a plane in very high - dimensional space ) that best separates the positive ( red ) and negative ( blue ) standards by gene expression .
unlabeled genes ( black ) are then ranked by their distance to this plane ( c ) , and the ranked list is returned to the user as predictions ( d ) .
the user is also provided detailed evaluation plots based on cross - validation ( figure 3 ) .
we pre - process separate datasets into a gene - expression compendium for each organism .
( b ) the user provides positive and negative standards ( either input by the user or from common databases ) and the data are labeled with these standards .
the svm algorithm identifies the maximum - margin hyperplane ( here a dotted line for the two - arrays in this example , but in practice this is a plane in very high - dimensional space ) that best separates the positive ( red ) and negative ( blue ) standards by gene expression .
unlabeled genes ( black ) are then ranked by their distance to this plane ( c ) , and the ranked list is returned to the user as predictions ( d ) .
the user is also provided detailed evaluation plots based on cross - validation ( figure 3 ) .
in addition to novel predictions , the user is provided with interactive visualizations of standard quantitative metrics for evaluating results of classification algorithms including the area under the curve ( auc ) , a figure showing the precision - recall trade - off , and a figure comparing the true positive rate and false positive rate ( shown in figure 3a ) .
pilgrm provides this high - quality results visualization with cross - platform javascript that is accessible without proprietary plugins in all modern web browsers .
javascript also allows for interactive figures that provide additional information on mouseover ( as with the mouseover display of genes from each standard shown in figure 3b ) .
this interactivity allows researchers to more fully understand how each gene in a standard is classified .
users also have the option of including validation standards that are also shown on this figure .
validation standards are not used for classification and can be used to highlight genes of interest or to further assess prediction quality .
all these results figures can be exported to jpg , png , svg or pdf for easy inclusion in reports and publications .
additionally , the web server is capable of producing a document for each analysis that provides a detailed explanation of the methods , data and results specific to a user s analysis .
this document is formatted as a pdf and is intended as supplementary data for molecular biology manuscripts informed by a pilgrm analysis . 

( a ) the true positive rate at various false positive rates for the case study of yeast dna - damage repair .
the area under the curve , shown in blue , is 0.7189 for this analysis and the performance of a random classifier is shown by the grey line .
( b ) illustrations how pilgrm figures are highly interactive . in this visualization , the rank and score from pilgrm are plotted for each gene in the positive ( red ) and negative ( green ) standards .
moving the mouse over a point shows which gene it represents . clicking on a standard
( a ) the true positive rate at various false positive rates for the case study of yeast dna - damage repair .
the area under the curve , shown in blue , is 0.7189 for this analysis and the performance of a random classifier is shown by the grey line .
the rank and score from pilgrm are plotted for each gene in the positive ( red ) and negative ( green ) standards . moving the mouse over a point shows which gene it represents .
clicking on a standard toggles it between shown and hidden ( here go:0006974 has been hidden ) .
we have evaluated other implementations ( including polynomial and rbf kernels ) and linear svm offers classification performance that is better or comparable to more complex forms often at substantially faster speed ( 16 ) .
pilgrm currently contains data and standards for six organisms ( human and the model organisms yeast , worm , mouse , rat and arabidopsis as detailed in table 1 ) and additional organisms are added upon request .
the data are processed uniformly and in a manner robust to diverse platforms and many experimental biases . as an example , for affymetrix data compendia all supplementary cel files available in the gene expression omnibus ( 17 ) are downloaded , and their probes mapped to entrez geneids using the entrez brainarray customcdf ( 18 ) .
all arrays are processed within their experiment ( geo series ) using the affy ( 19 ) r package from bioconductor ( 20 ) .
expression values are summarized with the medianpolish ( 21 ) method after rma background correction ( 21 ) and quantile normalization ( 22 ) . at this point ,
experiment sets with five or fewer arrays are combined into a single set of arrays .
genes are then normalized within experiments and combined for learning using our open - source c++ sleipnir library for computational functional genomics ( 23 )
each new analysis is assigned to the current organism - specific data compendium when it is created .
users can , at the granularity of individual analyses , elect to have pilgrm re - perform their exact analysis using the most current data compendium . because experimenters can include their own unpublished experimental results as part of their custom gold standards and because pilgrm predictions are used to direct follow - up bench experiments , pilgrm offers multiple levels of access control .
hidden analyses and standards are not shown in lists on the site and are accessible only through a user - defined web address .
this is the highest level of protection and prohibits access by anyone other than the analysis owner .
registration is simple , completely optional ( the only pilgrm capability that needs registration is making analyses completely private ) and requires only a username , working email address , and password .
optional example input , which builds a hypothetical analysis of genes relevant to breast cancer but not mammary tissue in general ( one step of this analysis is shown in figure 4 ) , is provided for these tutorials .
standards and analyses created during the tutorials can then be used outside of the tutorial framework . 

figure 4.pilgrm contains step - by - step tutorials that familiarize users with the system .
the optional example represents an analysis of breast cancer progression that avoids genes that appear relevant simply because they are expressed in mammary epithelium .
the optional example represents an analysis of breast cancer progression that avoids genes that appear relevant simply because they are expressed in mammary epithelium .
the pilgrm server is a flexible tool that biologists can use to develop data - driven predictions of gene properties directly relevant to their experimental questions in under an hour .
regularly updated data compendia and database - derived gold standards insure that pilgrm remains current .
its user - defined access control lets researchers include unpublished findings to iteratively improve prediction quality without compromising novel findings .
pilgrm gives expert biologists a chance to use their expertise to mine large scale genomic compendia quickly and easily .
this case study represents a researcher interested in identifying novel candidate genes that are involved in dna - damage repair while excluding genes only generally related to cell cycle control .
the first step of a pilgrm analysis is to determine what the positive and negative standards should be .
, the researcher uses a pilgrm - provided positive standard of yeast genes with direct experimental annotations to go:0006794 ( response to dna - damage stimulus ) and go:0000077 ( dna - damage checkpoint )
she elects to use a negative standard containing yeast genes with direct experimental annotations to go:0000278 ( mitotic cell cycle ) and go:0007049 ( cell cycle ) ; this standard is also included in pilgrm ( as are all go - based standards ) .
although in this case study the analysis uses only standards from the gene ontology s biological process ontology , researchers are free to customize these standards or add additional ones for their own analyses .
the researcher runs the analysis using pilgrm s yeast gene expression compendium , which consists of all s. cerevisiae expression ( gds ) datasets from geo .
the pilgrm data processing pipeline ( invisible to the user ) , has already done all the pre - processing for this analysis : the supplied probe identifiers were mapped to entrez identifiers ; each array was normalized with a fisher z - transform ; genes were normalized with experiments and combined for learning using our sleipnir library for computational functional genomics ( 23 ) . in total this compendium of s. cerevisiae gds datasets from geo contains 1801 arrays from 117 different experiments covering 6077 entrez gene identifiers as of 31 january 2011 .
she sees an auc visualization and is informed that the area under the curve for this analysis is 0.7189 ( figure 3a ) .
she also can examine the list of novel predictions , with link - outs to appropriate model organism databases to provide gene - specific information for each prediction .
in this case , the top novel prediction is the gene ymr090w , which sgd ( 24 ) lists as a putative protein with unknown function .
this gene is not essential ( 25 ) and is up - regulated in response to the fungicide mancozeb in a proteome - wide screen ( 26 ) .
mancozeb has been shown , in rats , to induce single strand breaks in a dose - dependent manner ( 27 ) .
thus , in this case study pilgrm discovers a potentially relevant gene not previously associated with dna - damage repair that has promising experimental support .
such analysis would take a researcher a total of 15  min to perform using pilgrm , including all analysis setup and definition of gold standards .
this complete analysis is available at http://pilgrm.princeton.edu / analysis / view / case - study - yeast - dna - damage - response/.
pilgrm is a user - friendly exploratory tool for expert biologists who wish to use current knowledge and genome - wide experimental data to guide the design of future experiments .
the extensive pre - loaded data collections and literature - based standards from common databases make it easy for researchers to start using the system .
pilgrm is being actively developed , and we will continue adding capabilities based upon user requests . currently we are working to include rna - seq data and developing an interface to allow users to perform an analysis on a user - defined subset of the data compendium .
this web server s flexibility allows biologists to customize analyses that address - specific questions of interest within diverse topics such as protein function , tissue - specific gene expression and cellular localization by employing computing approaches for data - driven generation of accurate hypotheses .
pilgrm thus brings sophisticated machine - learning methods applied to enormous gene expression compendia into the lab of any researcher , enabling data - driven experiment direction complementary to traditional knowledge - based discovery provided by existing databases .
national science foundation ( nsf ) career ( award dbi-0546275 ) ; national institutes of health ( grant r01 gm071966 ) ; national institute of general medical sciences ( nigms ) center of excellence ( grant p50 gm071508 ) ; national cancer institute ( nci ) inrsa t32 ca005928 . funding for open access charge :
 pilgrm ( the platform for interactive learning by genomics results mining ) puts advanced supervised analysis techniques applied to enormous gene expression compendia into the hands of bench biologists . 
 this flexible system empowers its users to answer diverse biological questions that are often outside of the scope of common databases in a data - driven manner . 
 this capability allows domain experts to quickly and easily generate hypotheses about biological processes , tissues or diseases of interest . 
 specifically pilgrm helps biologists generate these hypotheses by analyzing the expression levels of known relevant genes in large compendia of microarray data . 
 because pilgrm is data - driven , it complements a user s knowledge and literature analysis with mining of diverse functional genomic data , thereby generating novel predictions that can drive experimental follow - up . 
 this server is free , does not require registration and is available for use at http://pilgrm.princeton.edu . 
INTRODUCTION
SYSTEM DESCRIPTION
CASE STUDY: YEAST DNA-DAMAGE REPAIR
DISCUSSION
SUPPLEMENTARY DATA
FUNDING