like most large - scale 
 biological experiments , the vast majority 
 of proteomics studies tend to produce long lists of candidate proteins . 
 whether these consist of lists of interactors of a protein , differentially 
 expressed proteins , or simply proteome characterization of a sample , 
 such data sets are difficult to analyze and interpret .
no clear unifying 
 theory consistently explains the presence of all proteins or biological 
 processes in a given data set .
the picture 
 becomes even more intricate when large - scale protein quantification 
 is performed .
protein abundance measurements performed using label - free 
 or label - based methods contain a certain amount of noise , which contributes 
 to the complexity of proteomics data sets .
in addition , reproducibility of both protein identification and 
 quantification across biological replicates still remains an important 
 challenge . while it is expected that future 
 increases in mass spectrometry peptide detection sensitivity will 
 help solve reproducibility issues , it will cause data sets to be even 
 larger and therefore more convoluted .
computational approaches 
 are required to extract biologically meaningful 
 information from these complex data sets .
such tools are crucial in 
 order to get a clear understanding of the biological mechanisms taking 
 place in a given sample and are critical to guide further studies 
 on subsets of proteins within a data set .
multiple techniques were 
 proposed to analyze large - scale biological data sets , some of which 
 were originally applied to gene expression analysis but were then 
 adapted to proteomics studies .
a typical approach for the analysis 
 of a large biological data 
 set is to investigate the presence of enriched gene or protein sets 
 ( obtained from databases such as gene ontology ( go ) , molecular signature database ( msigdb ) , or kegg ) in a list of proteins 
 associated with a feature of interest ( e.g. , differentially expressed 
 genes , proteins with high abundance , etc . )
briefly , these methods use hypergeometric or fisher s 
 exact tests to compare the proportion of genes or proteins having 
 the feature of interest and belonging to the protein set associated 
 with an annotation a ( e.g. , go term ) to that of a in the complete data set of genes or proteins
. these tools 
 include , among others , ontologizer , fatigo , gostat , gorilla , and gominer .
such 
 techniques then assign a p - value representing the 
 probability that a greater or equal enrichment could be observed by 
 chance . although useful to provide an overview of the biological processes 
 and molecular mechanisms present in a given data set , these methods 
 suffer from several limitations .
most approaches assume that gene 
 or protein expressions in an organism are independent of each other 
 in order to assign enrichment p - values .
this is obviously 
 inaccurate as several genes are coregulated and coexpressed in the 
 vast majority of known organisms .
in addition , especially in the context 
 of proteomics , these methods consider protein membership in annotation 
 protein sets as binary and do not take into account protein quantification 
 measurements , such as spectral counts or relative abundances .
furthermore , 
 arbitrary thresholds based on the feature of interest are required 
 ( e.g. , minimum protein spectral count ) in order to determine which 
 proteins should be tested for annotation enrichment .
this causes two 
 genes or proteins , where one has a value slightly above the threshold 
 and the other a shade lower , to be treated strikingly differently 
 although they behave extremely similarly .
an alternative approach 
 utilizes the popular gene set enrichment 
 analysis ( gsea ) , a computational tool originally 
 intended for gene expression analysis using microarrays . unlike classic 
 go enrichment analyses , gsea does not require an arbitrary threshold 
 to determine the subset of genes with the feature of interest , but 
 instead ranks all the genes in a given data set based on a feature 
 of interest , such as the level of differential expression between 
 two conditions .
it then assigns a weighted enrichment score es to 
 an annotation gene set ( derived from the msigdb ) based on its clustering toward the top or bottom of that 
 ranked ordered gene list .
finally , gsea assesses the statistical significance 
 of the es and assigns it a p - value .
gsea was implemented 
 in various software packages including gsea - p and web - based tools such as genetrailexpress .
while gsea addresses several issues of the previously presented 
 go enrichment analysis scheme , it still suffers from important drawbacks . 

however , it requires the contrast of two classes or 
 conditions ( e.g. , disease vs normal or mutant vs wild type ) in the 
 analyzed data set in order to be executed .
this feature is reasonable 
 for the intended input of gsea ( i.e. , microarray gene expression ) , 
 but is a major limitation in the context of proteomics data sets because 
 many proteomics experiments are designed to characterize the proteome 
 of a single organism or to identify a set of protein interactors without 
 comparing the detected proteins to those found in another condition .
these include a dynamic programming algorithm to calculate the exact 
 statistical significance of a modified unweighted es .
the same approach can be used to perform an unweighted 
 gsea analysis on single class data sets , while assuming gene expression 
 independence .
jiang et al . published 
 an extension of gsea that takes into account annotation gene set overlaps .
in addition , this extension proposed different 
 statistics to measure the level of association between a gene set 
 and the feature of interest .
a user - friendly 
 stand - alone software tool erminej also 
 used a permutation based procedure to evaluate gene set enrichments . 

finally , gsea was applied to a proteomics data set related to dilated 
 cardiomyopathy . since gsea was 
 originally introduced to analyze gene expression 
 data originating from microarrays , an adapted version tailored to 
 proteomics data sets called psea was 
 proposed by cha et al .
like gsea , it does not assume protein expression 
 independence , and still relies on the presence of two conditions in 
 the analyzed data set to identify annotation gene sets for which the 
 proteins are significantly differentially expressed between the two 
 classes .
psea measures protein differential expression using the spectral 
 counting based measure called spectral index ( spi ) .
an important difference between microarray gene expression 
 data and mass spectrometry protein abundance measurements , such as 
 spectral counts , resides in the technical variability of both approaches . 

this variability is greater and more difficult to account for in protein 
 mass spectrometry .
for example , equally 
 abundant peptides may not have the same likelihood of being detected 
 using mass spectrometry , and therefore yield different abundance measurements . 

in the context of liquid chromatography coupled to data - dependent 
 mass spectrometry analysis , detection and quantification of a given 
 peptide is dependent on the abundance of other coeluting peptides . 

equally abundant rna fragments are equally 
 likely to hybridize to their complementary strand on a microarray 
 if the corresponding probes are present .
the spi attempts to capture 
 this protein abundance variability in a rather crude method by weighting 
 a given protein spectral count by the fraction of experiments in the 
 condition in which the protein was detected .
a protein detected with 
 high spectral counts in all experiments and another detected with 
 high spectral counts in half of the experiments and low ones in the 
 other half would therefore be considered to have the same variability 
 according to the spi .
furthermore , psea was only designed to take 
 spectral counts as input and can not be used to analyze data sets obtained 
 with label - based techniques of protein quantification ( such as silac , itraq , tmt , and n ) .
in addition to these limitations , there is currently no user - friendly 
 implementation of psea that is publicly available . in this paper , 

we propose a novel user - friendly , publicly available , 
 online protein set enrichment analysis tool ( psea - quant ) that allows 
 the analysis of protein quantification data sets with replicated experiments 
 composed of samples derived from a single or multiple conditions . 

psea - quant statistically assesses the enrichment of proteins with 
 high abundance and well reproduced abundance measurements across a 
 set of replicates in protein sets from go and msigdb .
such protein 
 sets are of interest because they represent proteins with highly reliable 
 abundance measurements of great amplitude that are likely to be linked 
 to the phenotype of the sample .
psea - quant uses a permutation scheme , 
 which models both protein abundance dependencies and annotation biases . 

we show that psea - quant works with both label - free and label - based 
 protein quantification methods and yields complementary results to 
 classic go enrichment analysis tools ( gorilla and ontologizer ) .
when psea - quant was 
 applied to a label - free quantified cystic fibrosis data set and to 
 a label - based quantified human , mouse , and rat brain frontal cortex 
 data set , our approach highlighted several processes and mechanisms 
 that are of putative biological interest and are linked to observed 
 phenotypes .
the goal of our approach is to identify protein 
 sets , such as go 
 terms or those available in the msigdb , that are significantly enriched for proteins 
 with high abundance measurements and that are reproducible across 
 replicates in both label - free and label - based quantification data 
 sets . to this end
, we present a protein enrichment score , pes , and assign one to all protein sets including at least 
 two proteins in the analyzed data set .
we then assess the statistical 
 significance of all pes using monte carlo sampling 
 procedures and evaluate a false discovery rate using a permutated 
 data set .
a greedy algorithm is finally applied to identify the core 
 of all significantly enriched protein sets .
the first one consisted of label - free quantified samples using protein 
 spectral counting of lung epithelial cell lines stably expressing 
 a cftr mutant ( cfbe ) or the wild type cftr ( wild type ) . both cfbe 
 and wild type proteomics analyses were performed in biological triplicates . 

the second data set is composed of two separate tandem mass tag ( tmt ) 
 labeled protein quantification analyses of brain frontal cortices 
 of human , mouse , and rat ( hmr ) and of human and rat ( hr ) .
the hmr 
 and hr data sets consist of technical and biological replicates , respectively . 
 the details of the sample preparation and mass spectrometry data acquisition 
 for the above data sets are provided in the supporting 
 information .
protein spectral counts were obtained for 
 all proteins in both the cfbe and wild type data sets .
spectral counts 
 were normalized for each replicate by the sum of all spectral counts 
 within the replicated experiment .
we first applied psea - quant on the 
 label - free quantified cfbe samples .
we then show that psea - quant can 
 be extended to analyze a label - free quantified data set comprising 
 two contrasting conditions using both cfbe and wild type samples , 
 even though it is not its primary purpose
. for the hmr and hr 
 data sets , protein relative abundances were 
 calculated by dividing the normalized intensity of a protein for a 
 tmt sixplex label in one condition ( e.g. , human ) by the normalized 
 intensity of the same protein for a tmt sixplex label in another condition 
 ( e.g. , mouse ) ( see the supporting information for protein normalized intensity calculation ) .
these protein normalized 
 abundance ratios were computed for all pairs of protein normalized 
 intensities .
for example , in the context of the hr data set , all ratios 
 corresponding to the tmt sixplex labels 126/129 , 126/130 , 126/131 , 
 127/129 , 127/130 , 127/131 , 128/129 , 128/130 , and 128/131 were computed . 
 these correspond to replicate measurements of the expression of human 
 proteins over rat proteins . in this example , the inverse ratios can 
 be taken as a measure of the expression of rat proteins over human 
 ones . unlike for label - free quantification where psea - quant s 
 main goal is to analyze single condition data sets ,
psea - quant was 
 explicitly designed to handle label - based protein quantification data 
 sets containing multiple conditions and was therefore applied to the 
 hmr and hr data sets .
psea - quant 
 is composed of four 
 main steps , which are illustrated in figure 1 : ( 1 ) protein enrichment score ( pes ) computation 
 for each protein set .
( 2 ) statistical significance assessment of pes . ( 3 ) false discovery rate estimation .
psea - quant computes 
 a pes for each protein set and assesses the statistical 
 significance of 
 each pes by providing a p - value . 

these p - values are then transformed to q - values to correct for multiple hypothesis testing .
for 
 each protein set , 
 psea - quant computes an enrichment score , which reflects the abundance 
 and reproducibility of the abundance measurements of the proteins 
 in the protein set .
if the data set under study is quantified using 
 a label - free approach , psea - quant takes the protein spectral counts 
 as input and then normalizes them as previously described . on the 
 other hand ,
if the data set is quantified with a label - based approach , 
 such as itraq , tmt , silac , or n , psea - quant uses the 
 protein relative abundances quantification values ( protein normalized 
 abundance ratios ) .
psea - quant calculates the pes of 
 a protein set by considering the mean abundance and abundance coefficient 
 of variation across replicates for all proteins in the set . the abundance 
 coefficient of variation reflects the reproducibility of abundance 
 measurements for a given protein .
it was chose since it allows the 
 comparison of sets of abundance values of proteins that possess widely 
 different means .
figure 2 illustrates 
 the pes calculation of a hypothetical protein set 
 in the context of the cfbe data set .
the protein mean abundance   = ( i  =  1rap , i)/r corresponds to the 
 mean normalized spectral count of a 
 protein p across all i = 1 ... r replicates ( ap , i ) for label - free quantification . on the other hand , for label - based 
 quantification ,  
is equal to the mean normalized abundance 
 ratio of p across all r replicates . 

let cvp = p/ be the coefficient of variation 
 of p , where p = 
 [ ( 1/r)i  =  1r(ap , i  ) ] . 

to ease the pes calculation , coefficients of variation 
 are transformed 
 to be on the same scale as abundance values using the following equation : where p is the set of all 
 proteins in the analyzed data set .
this scaling function preserves 
 the ratios and relative differences between all pairs of coefficient 
 of variation values in the data set . both   and cvpt values are then discretized 
 appropriately into integers .
psea - quant 
 calculates the pes of a protein set s by mapping proteins p according to their mean abundance   ( label - free or label - based ) and 
 their coefficient 
 of variation of the abundance cvpt values onto a n by n enrichment 
 score weight matrix w , whereand n = maxpp .
the previous equation corresponds 
 to filling w with values from the bottom left corner 
 ( high cvpt and low ) to the top right corner ( low cvpt and high ) in a diagonal fashion with values 
 increasing 
 from 0 to 1 with a constant increment . based on its position in w ,
proteins with high abundance mean and low coefficient 
 of variation therefore obtain high weights .
weights are then summed 
 for all proteins of a protein set s that are present 
 in the analyzed data set to provide s with a pes.all protein sets originating 
 from either the 
 go or msigdb database containing at least two proteins that are observed 
 in the analyzed data set are assigned a pes . unlike 
 gsea and psea , calculation of this score ignores proteins in the analyzed 
 data set that are not present in the protein set being evaluated . 

as an alternative to the pes , a t - statistic could have been used to compare individual protein 
 mean abundance and standard deviation of abundance values to the average 
 mean abundance values and average standard deviation of abundance 
 values in the entire data set .
we chose not to use this approach since 
 our goal was to equally weight the reproducibility of abundance measurements 
 and the average abundance values .
the go database used in this study 
 includes 17  214 protein sets containing on average 118.95 proteins 
 ( december 19 , 2013 ) , while the msigdb comprises 8841 protein sets 
 ( not including go terms ) , with an average size of 132.43 proteins 
 ( march 11 , 2014 ) .
the actual size of a protein set being scored varies 
 across the different data sets analyzed , since only proteins observed 
 in the experimental data set are considered for the pes calculation . the average protein set sizes tested for the cfbe data 
 set and the wild type data set were 53.40 ( 8255 tested protein sets ) 
 and 52.93 ( 8230 tested protein sets ) , respectively , for the go database 
 and 51.30 ( 8519 tested protein sets ) and 50.57 ( 8488 tested protein 
 sets ) for the msigdb .
similarly , the average protein set size was 
 42.23 ( 7919 tested protein sets ) for the hmr data set and 42.86 ( 8061 
 tested protein sets ) for the hr data set using the go database .
( a ) heat map representations of log 
 normalized spectral counts log(sc ) and normalized spectral count coefficients 
 of variation ( cv ) for all proteins identified in 
 the three biological replicates of the cfbe data set .
a fictitious 
 example of an annotation protein set of seven proteins is illustrated 
 on the heat maps .
( b ) color - coded representation of the enrichment 
 score weight matrix w. proteins p from the fictitious protein set in ( a ) are mapped onto w based on their respective protein mean abundances   and abundance coefficients of 
 variation cvpt .
the sum of the 
 enrichment 
 score weights corresponding to their position in the matrix is then 
 assigned as the pes of the fictitious protein set .
the statistical 
 significance of the pes of a protein set s is computed using 
 a monte carlo sampling procedure .
multiple alternative sampling procedures 
 can be employed for this computation , each with their advantages and 
 drawbacks .
the first monte 
 carlo procedure called  uniform sampling  repeatedly 
 samples a subset of |s| proteins in the analyzed 
 data set , with each protein having an equal probability of being randomly 
 sampled , and computes the pes for each of these random 
 subsets to obtain an unbiased estimate of the pes distribution for |s| proteins . from this distribution
, 
 the p - value of a pes of a random 
 subset can be computed representing the probability that it is larger 
 or equal to the pes of s. although the uniform 
 sampling method is unbiased , it ignores that proteins of high abundances 
 are often those that have the most protein annotations and therefore 
 belong to more protein sets .
this is in part caused by the fact that 
 these proteins are frequently those that are the most discussed in 
 the literature , and also simply because some proteins perform more 
 functions and are involved in more biological processes than others . 

this protein annotation bias could lead to an underestimation of protein 
 set p - values , as heavily annotated proteins of high 
 abundance will not be randomly selected as many times as they should . 
 to correct this issue
this alternative method samples |s| proteins in the data set with a probability proportional 
 to their respective number of occurrences in the ensemble of protein 
 sets in the database ( go or msigdb ) being tested for enrichment .
this 
 approach models protein annotation biases and ensures more accurate p - value estimations , but could be slightly slower for large 
 protein sets when the protein sampling probability distribution deviates 
 greatly from the uniform distribution .
these two sampling procedures implement a random 
 selection of proteins in order to obtain the null distribution of pes .
these methods assume among other things , that the abundances 
 of proteins within the same protein set are independent .
this differs from the approach used 
 by gsea and psea , which avoids making such assumption and takes advantage 
 of class label permutation ( disease vs normal condition ) in order 
 to perform statistical assessments . as mentioned earlier , since proteomics 
 data sets
are often composed of experiments originating from a single 
 condition , such permutation scheme is not always applicable .
we therefore 
 introduce a novel strategy , which consists of randomly selecting proteins 
 using either the weighted or uniform sampling method in a reduced 
 sample space . in an attempt to model protein abundance dependencies
, 
 we repeatedly sample |s| proteins at random and build 
 the null distribution of pes using only samples for 
 which the coefficient of variation of the mean abundances   of sampled proteins falls within 
 a user - defined 
 range . this range is defined by applying a tolerance factor on the 
 coefficient of variation of the mean abundances   of the 
 proteins in the protein set s , which is being statistically 
 assessed . specifically
, 
 let c be the coefficient of variation of the s of the protein set s being statistically assessed and let 0 < f  
 1.0 be the coefficient of variance tolerance factor ( cv tolerance ) .
let c be the coefficient of 
 variation of the mean abundances of a randomly sampled protein set s. then ,
if f c  c  ( 1 + f)c , sample s will be used to build the null 
 distribution of pes . of note , this coefficient of 
 variation c of the mean abundances   of the proteins within a randomly 
 sampled 
 protein set is different from the coefficient of variation ( cvp ) computed in the pes , which calculates the variation of the abundance measurements 
 of a single protein across the replicate experiments .
the reasoning 
 behind this approach is that proteins in the same protein set are 
 more likely to have similar abundances than randomly selected proteins 
 ( e.g. , proteins that are members of the same protein complex ) , and 
 therefore , protein abundances within such protein sets are more likely 
 to yield a small coefficient of variation than a random set of proteins . 
 while this approach does not model all protein dependencies within 
 a protein set , it tackles an important issue and is likely to provide 
 a more realistic estimate of the true distribution of pes .
it should be noted that the sample space reduction by the cv tolerance can be applied to both the uniform and weighted 
 sampling procedures presented above .
obviously , obtaining enough samples 
 to produce a reliable null distribution of pes under 
 this sampling method may require a longer running time depending on 
 the cv tolerance that is selected by the user . since 
 databases such as go and msigdb contain a large number of protein 
 sets , multiple hypothesis testing correction of the above computed p - values is required . because of the important dependencies 
 and overlaps between the various protein sets , the computation of 
 an empirical q - value ( false discovery rate adjusted p - value ) for each p - value is more appropriate 
 than a too stringent bonferroni correction . to estimate the empirical q - value for each protein set p - value , we 
 randomize all protein sets and compute the protein set enrichment p - values of these random protein sets using steps ( 1 ) and 
 ( 2 ) ( see methods ) .
let n(p ) be the number of nonrandomized 
 protein sets that obtained a p - value of at most p and r(p ) be the number 
 of randomized protein sets that are also associated with a p - value of at most p. we compute the false 
 discovery rate ( fdr ) for all protein sets with a p - value  p as fdr(p ) = r(p)/n(p ) .
fdr(p ) may 
 not be monotonic since it independently computes a fdr for each p - value threshold .
such adjustment was proposed by yekutieli and benjamini to adjust p - values that were corrected for multiple hypothesis testing and was applied in similar contexts . finally , for each protein set s with a p - value ps , 
 the empirical q - value is equal to fdr(ps ) .
a protein set s for which the proteins 
 are significantly abundant and have a surprisingly small abundance 
 coefficient of variation will obtain a small p - value 
 and q - value .
it does not , however , mean that all 
 of the proteins in that protein set are abundant and have a small 
 abundance coefficient of variation , but simply that a significant 
 subset of s does .
the algorithm iteratively removes 
 from s the protein for which the removal improves 
 the p - value the most ( i.e. , protein with the minimum 
 enrichment score weight ) until no further p - value 
 improvements can be performed .
this greedy algorithm does not guarantee 
 optimality but generally succeeds at identifying the key component 
 of s. the proposed computational 
 tools are implemented in a platform - independent java program called 
 psea - quant . to decrease running time ,
psea - quant first performs a 
 presampling procedure where it estimates the p - value 
 of a given protein set using a small number of random samples ( 100 ) . 

if this p - value is less than 0.02 , then the p - value estimation is refined using the user defined number 
 of random samples .
psea - quant can be used on the web in a user - friendly 
 fashion at krusty.scripps.edu:8080/psea-quant or downloaded 
 and used as a command - line tool at krusty.scripps.edu:8080/psea-quant/files .
the cfbe and human vs rat frontal cortex ( hr ) data sets are provided 
 as sample input files at this address : krusty.scripps.edu:8080/psea-quant/files .
psea - quant is an approach 
 that identifies protein set annotations 
 that are enriched for proteins with high abundance and low abundance 
 measurement variation . in this paper , we applied psea - quant using 
 go and msigdb protein sets to a label - free quantified protein data 
 set related to cystic fibrosis ( cfbe ) and to an isobaric labeling - based 
 tmt quantified protein data sets of the frontal cortex of human and 
 rat ( hr ) and human , mouse , and rat ( hmr ) brain . while tmt was used 
 as an example of label - based quantification that can be used in conjunction 
 with psea - quant , it could have been replaced by any one of a number 
 of other approaches such as itraq , silac , or n. psea - quant 
 uses a variety of approaches to estimate the statistical significance 
 of protein sets in a given data set and assigns them p - values .
we validated these approaches by showing that the p - value distributions obtained with all techniques are largely 
 uniformly distributed under the null hypothesis ( see supporting information figure s1 and text for complete analysis ) .
since a large number 
 of protein sets are being tested for their enrichment ( e.g. , 8255 
 go terms that were associated with at least two proteins in the cfbe 
 data set ) , psea - quant computes an empirical q - value 
 for each protein set p - value .
the results presented 
 in supporting information figure s1 highlighted 
 that the protein number randomization technique is 
 the most conservative randomization scheme and maintains protein annotation 
 biases .
it was therefore selected to estimate the false discovery 
 rate and q - values in psea - quant results .
figure 3 shows the fdr for small go protein set p - values (  0.01 ) obtained for the cfbe data set . 
 there is a striking difference between the fdrs obtained for the same p - values between the uniform and weighted monte carlo sampling 
 approaches .
however , with both uniform and weighted sampling approaches , 
 only the curves corresponding to a low cv tolerance 
 ( 0.1 ) seem to behave slightly differently from the other ones obtained 
 with the same sampling approach .
figure 3 demonstrates 
 that for both sampling approaches , using a low cv tolerance is the most conservative approach , since it yields the 
 highest fdr for the vast majority of p - values .
the 
 same observations can be made on the fdr curves of the wild type data 
 set ( supporting information figure s2 ) . 
 since we have described that the weighted monte carlo sampling technique 
 models protein annotation biases and is therefore more likely to yield 
 biologically significant results , and that a low cv tolerance ( 0.1 ) is the most conservative approach , these strategies 
 are the ones that we used for the remaining data analyses in this 
 article .
fdrs associated with enrichment p - values computed 
 by psea - quant for the cfbe data set .
the p - values 
 were computed using the uniform and weighted monte carlo sampling 
 procedures with mean abundance cv tolerance values 
 of 1.0 , 0.5 , and 0.1 of the sampled protein sets ( as described in methods ) .
when no cv tolerance 
 was applied , the data are labeled as  protein independence  . 

we compared 
 the go protein sets identified by psea - quant in the cfbe data set 
 ( see supporting information file s1 for 
 complete results ) to those found to be enriched according to the go 
 enrichment analysis tools gorilla and 
 ontologizer .
one could choose to test for 
 the over - representation of a go protein set in the entire data set 
 based on what is expected in the gene ontology database .
another option 
 is to set abundance and abundance coefficient of variation thresholds 
 and test for the enrichment of go protein sets within the subset of 
 proteins passing these thresholds ( proteins with the feature of interest ) , 
 while using the entire data set as background .
we ran gorilla using 
 the entire data set as a ranked protein list based on protein abundance 
 and abundance coefficient of variation across replicates .
we also 
 executed gorilla with thresholds on the protein abundance and abundance 
 coefficient of variation across replicates so that the top 10% proteins 
 were selected as having the feature of interest and the complete data 
 set was considered as the background subset .
the best p - value among the two gorilla runs was considered for the benchmarking 
 analysis ( see supporting information files 
 s2 and s3 for complete results ) .
in addition , we executed ontologizer 
 using the same top 10% protein subset and considering the complete 
 data set as background .
we also performed an ontologizer analysis 
 independently on each cfbe replicate using the entire go database 
 as background .
we then associated the best p - value 
 from all ontologizer runs to each go protein set ( see supporting information file s4 and s5 for complete 
 results ) .
table 1 shows the go terms for which 
 the protein sets were found to be significant according to psea - quant 
 ( q - value < 0.1 ) , but not by using the gorilla 
 and ontologizer analyses ( p - value  0.001 ) . 

interestingly , an important number of go terms found by psea - quant 
 and not by gorilla and ontologizer are closely related to cystic fibrosis . 
 indeed , among the go terms of interest , we find that proteins annotated 
 with  glucose transport  ( p - value = 
 2.0  10 and q - value = 0.02 ) 
 and  hexose transport  ( p - value = 2.5 
  10 and q - value = 0.03 ) 
 are significantly abundant and that their abundance is measured with 
 a surprisingly low coefficient of variation .
these two biological 
 processes have been shown to be disrupted in cystic fibrosis patients , and glucose transport in particular is enhanced in various tissues 
 of cystic fibrosis patients .
the leading cause of 
 cystic fibrosis is the misfolding of cftr caused by a deletion of 
 f508 of the cftr gene .
it is therefore not without interest 
 that we observe that psea - quant highlights go terms , such as  protein 
 binding involved in protein folding 
( p - value 
 = 6.0  10 and q - value = 
 0.01 ) and  misfolded protein binding 
( p - value = 9.9  10 and q - value = 0.09 ) , to be significant .
it is also well described that 
 sodium channels and chloride channels are disrupted in cystic fibrosis ; for instance , airway epithelia in cystic fibrosis patients have 
 a higher rate of sodium absorption than normal . hence , to find  response to salt stress  

( p - value = 2.3  10 and q - value = 0.03 ) among the go protein sets identified by 
 psea - quant is in accordance with the literature .
evidently , psea - quant 
 finds some go protein sets that do not include a large number of proteins 
 ( see table 1 ) .
however , these proteins are 
 more abundant and their abundance measurements have a lower coefficient 
 of variation than expected by chance .
both gorilla and ontologizer 
 find several significantly enriched go protein sets in our data that 
 psea - quant does not .
this is expected since a go protein set may be 
 enriched in terms of number of occurrences in a data set , but not 
 enriched for proteins of high abundance with well reproduced abundance 
 measurements .
the goal of psea - quant is not to outperform tools like 
 gorilla or ontologizer but rather to complement them .
these differences 
 in the results highlight the complementarity of the two strategies , 
 where one analyzes the overrepresentation of a protein set in the 
 data ( go analysis ) , and the other ( psea - quant ) focuses on features 
 of interest ( e.g. , abundance , abundance coefficient of variation , 
 etc . ) of the proteins associated with a protein set .
redundant go terms were removed 
 and p - values and q - values were rounded .
as shown previously , psea - quant found various significant 
 go protein sets that were relevant to cystic fibrosis when simply 
 provided the cfbe data set as input .
psea - quant is capable of handling 
 such a label - free quantified data set composed of a single condition , 
 but it can also be extended to use a data set that includes a contrasting 
 condition to produce biologically relevant results , even though it 
 was not explicitly designed for this task .
table 2 shows the most significant go protein sets using psea - quant 
 in the cfbe data set ( q - value < 0.06 ) but not 
 in the wild type data set ( q - value  0.1 ) . 

the vast majority of the go protein sets discussed above was found 
 to be significant in the cfbe data set ( q - value < 
 0.1 ) and not in the wild type , with the exception of  glucose 
 and hexose transport  , which was also significant in the wild 
 type data set .
it should be noted that these supplementary go protein 
 sets were also found to be significant by gorilla or ontologizer . 
 among these
( p - value = 1.0  10 and q - value < 0.01 ) , which is very tightly involved in the 
 degradation of cftr .
( p - value = 1.0  10 and q - value < 0.01 ) was also 
 found as significant only in the cfbe data set .
this result is consistent 
 with the literature , which indicates that cystic fibrosis patients 
 have an increased sensitivity to the cytokine interleukin-4 . on the other hand ,
supporting 
 information table s1 shows the most significant go protein 
 sets in the wild type data set ( q - value < 0.03 ) 
 ( see supporting information file s6 for 
 complete results ) that are not in the cfbe data set ( q - value  0.1 ) . among the go terms of interest ,
we find  cell 
 redox homeostasis  ( p - value < 10 and q - value < 0.01 ) .
indeed , calcium homeostasis 
 is known to be abnormal in cystic fibrosis airway epithelial cells .
the presence of the go term  actin cytoskeleton  
 is also noteworthy since actin was reported to bind and potentially 
 affect the functional properties of cftr .
the results in table 2 and supporting information table s1 highlight that psea - quant 
 identifies protein sets that are of putative biological interest with 
 regards to cystic fibrosis . of note , it is not because the relationship 
 between certain go terms listed in table 2 and supporting information table s1 and cystic fibrosis 
 is not obvious that these terms are irrelevant . these protein sets 
 might prove themselves useful in order to understand the differences 
 in the proteome between the cfbe and wild type samples .
figure 4 shows representative examples of go terms and their protein 
 sets that are related to cystic fibrosis for which the protein sets 
 were classified as significant according to psea - quant for the cfbe 
 data set but not the wild type one .
scatter plot of the union of all proteins 
 identified in all three 
 replicates of the cfbe data set .
the normalized spectral count coefficient 
 of variation across all three replicates of each protein is plotted 
 against its mean normalized spectral count .
proteins annotated with 
 representative examples of go terms related to cystic fibrosis identified 
 as significant by psea - quant in the cfbe data set , but not in the 
 wild type one are color - coded .
hspa1a and hsp90ab1 are respectively 
 colored in brown and purple due to their presence in more than one 
 protein sets .
protein 
 names in bold correspond to proteins belonging to the core of a protein 
 set .
redundant go terms were removed 
 and p - values and q - values were rounded .
psea - quant is not limited to the use of protein 
 sets originating from the go database .
indeed , any type of protein 
 set from kegg pathways to user - defined 
 protein sets can be used as input for psea - quant . to demonstrate this 
 feature of our tool , we used psea - quant to identify msigdb protein 
 sets that were significant in the cfbe ( q - value < 
 0.1 ) but not in the wild type data set ( q - value  
 0.1 ) ( see supporting information file s7 ) . 
 among the protein sets identified by psea - quant ,
 ( p - value = 9.6  10 and q - value = 0.07 ) , which contains genes that are up - regulated under 
 glucose - deprived conditions .
this result is notable since , as discussed 
 earlier , glucose transport is disrupted in cystic fibrosis cases . 

this analysis also allowed us to discover additional protein sets 
 of interest that are related to cystic fibrosis .
( p - value = 1.5  10 and q - value = 0.02 ) was found significant .
this protein set 
 includes genes involved in signaling events mediated by hdac class 
 ii .
interestingly , hdac7 , a hdac class ii protein , is known to play 
 a role in the restoration of the function of f508cftr .
in addition , psea - quant identified the kegg 
 protein set :  glutathione metabolism 
( p - value = 3.1  10 and q - value = 0.03 ) .
this observation is noteworthy since cystic fibrosis 
 patients are known to be glutathione deficient .
camp up.v1 up  
 ( p - value = 3.0  10 and q - value = 0.01 ) , which contains genes that are up - regulated 
 in response to camp signaling pathway activation , was also found to 
 be significant in the cfbe data set .
this result is interesting since 
 cftr is a camp - activated atp - gated anion channel .
psea - quant 
 was applied to both the human - rat ( hr ) and human - mouse - rat ( hmr ) tmt 
 label - based quantification data set .
mice and rats are commonly employed 
 to mimic neurological disorders due to the amazing conservation of 
 the brain across mammalian species .
differences , however , do exist 
 and understanding them can lead to improved animal models of diseases . 

go protein sets identified as significantly up - regulated with a low 
 coefficient of variation in human vs rat in the hr data set are shown 
 in table 3 ( q - value < 0.01 , 
 see supporting information file s8 for 
 complete results ) .
table 3 shows an important 
 number of very significant go terms related to mitochondria such as 
  nadh dehydrogenase activity  ,  mitochondrion  , 
  mitochondrial atp synthesis coupled proton transport 
mitochondrial 
 electron transport , nadh to ubiquinone  . obviously , while some 
 are mostly independent ,
not all the protein sets associated with these 
 go terms are mutually exclusive . however , an important number of proteins 
 related to mitochondria are significantly up - regulated in human vs 
 the rat .
mitochondria are the main source of atp in the brain but 
 also possess other functions such as calcium homeostasis and regulation 
 of apoptosis .
the atp powers synaptic activity , 
 ionic membrane gradients , propagation of action potentials , and axonal 
 transport .
these results may potentially 
 be explained by the fact that even if the rodent brain was the same 
 size of the human , it would still possess only an eighth of the neurons 
 in the human cerebral cortex .
therefore , 
 this would also argue for a higher number of mitochondria in the human 
 frontal cortex compared to rat to support the greater neuronal activity 
 of the human brain .
we observe a similar trend when analyzing with 
 psea - quant protein sets significantly up - regulated with a low coefficient 
 of variation in human vs mouse in the hmr data set ( table 4 ) . of note ,
the results are extremely similar when 
 performing the same analysis of human vs rat in the hmr data set ( data 
 not shown ) .
these results are potentially interesting for many human 
 specific neurological diseases , such as alzheimer s disease , 
 parkinson s disease , schizophrenia , and depression , that are 
 in part due to mitochondria dysfunction .
redundant 
 go terms were removed 
 and p - values and q - values were rounded .
redundant go 
 terms were removed 
 and p - values and q - values were rounded .
in contrast , when looking for 
 go protein sets that are significantly 
 upregulated with low coefficient of variation in the mouse and rat 
 frontal cortices when compared to human , fewer go protein sets were 
 identified by the psea - quant analysis of the hmr and hr tmt data sets 
 ( see supporting information tables s2 and 
 s3 ) .
interestingly , although not extremely significant , both rodents 
 when compared to human appear to have an overexpression of synapse 
 and dendrite related proteins .
it is worth noting that it is difficult 
 to supply an explanation for certain go terms identified by psea - quant 
 in this context since , to our knowledge , such protein quantification 
 experiments across closely related species have never been performed . 

hence , some of the go protein sets identified 
 by psea - quant in supporting information tables s2 and s3 could be of putative biological interest .
also , 
 as expected because of their evolutionary conservation , the analysis 
 did not reveal striking differences in terms of go protein set enrichments 
 between the mouse and rat frontal cortices ( see supporting information tables s4 and s5 ) , with the only major 
 difference being that a number of laminins and collagens appear up - regulated 
 in rat vs mouse .
these cause some laminins and collagen related go 
 protein sets to obtain significant p - values according 
 to psea - quant .
psea - quant 
 highlights protein sets of putative biological relevance 
 that are not identified using classic go enrichment analysis software 
 packages .
we believe that , when used in conjunction with such tools , 
 psea - quant facilitates the analysis of complex protein quantification 
 data sets . as mentioned previously , psea - quant uses replicated 
 experiments 
 in order to identify protein sets of interest .
a psea - quant analysis 
 requires a minimum of two replicates to compute an abundance coefficient 
 of variation for each protein .
obviously , if more replicates are given 
 as input to psea - quant , it improves its capacity to differentiate 
 proteins with different abundance variations .
our results show that 
 three replicate label - free experiments were sufficient to obtain results 
 of putative biological interest . as for
isobaric labeled experiments , 
 when two conditions were labeled with , two isobaric labels each ( hmr 
 data set ) , the resulting four abundance ratios , when used as input 
 to psea - quant , led to interesting results .
interestingly , the comparisons 
 of the human and rat samples obtained similar psea - quant results whether 
 two ( hmr ) or three ( hr ) replicate samples for each species were labeled .
the nature of the replicates in a data set , biological or technical , 
 is important to consider and affects the interpretation of the results 
 of psea - quant . when provided technical replicates as input , the pes computation of psea - quant measures the mean abundance 
 and the reliability of the abundance measurements of a protein with 
 the coefficient of variation .
therefore , in this context psea - quant 
 identifies protein sets enriched for proteins of high abundance values 
 measured with great reliability . when biological replicates
are used 
 as input to psea - quant the pes reflects both the 
 reliability of the abundance measurements and also the biological 
 variability .
therefore , psea - quant highlights protein sets containing 
 proteins with surprisingly high and constant abundance whose abundance 
 values are measured reproducibly across the biological replicates . 
 in this scenario , one could expect matrix proteins such as actins 
 to be among the top results . even though this can be the case ,
our 
 results clearly demonstrate that our method is sensitive enough to 
 detect protein sets , which might be of greater biological interest . 
 nevertheless , the identification of protein sets containing proteins 
 of high abundance with low abundance coefficient of variation is crucial 
 to try to understand the biological processes taking place in a sample 
 whether the sample is derived from an affinity purification , a whole 
 cell extract , or a bodily fluid .
in addition , the presence of matrix 
 proteins among the top results can only arise when a single condition 
 label - free quantified data set is given as input to psea - quant . when 
 samples that are quantified using a label - based approach or when a 
 data set including two different conditions are given as input , matrix 
 proteins tend to not appear in the results due to a lack of differential 
 expression .
furthermore , this problem also occurs for the vast majority 
 of go analysis software packages currently available .
although 
 only spectral counts and tmt - labeled protein abundance 
 ratios were used as input for psea - quant in this paper , our approach 
 can easily be applied to other protein quantification techniques such 
 as itraq , silac , and n. all of these methods perform 
 a relative quantification of peptides and , therefore , proteins .
protein 
 abundances obtained by these quantification approaches can be used 
 to compute pes , the same way tmt protein abundance 
 ratios were used in the present analysis .
psea - quant is also applicable 
 to label - free and label - based protein quantification of proteins obtained 
 through affinity purification coupled to mass spectrometry .
our software 
 package can be particularly useful in such context to characterize 
 the protein  protein interactions of a given protein .
in addition , we did not perform extensive noise processing 
 of the analyzed data sets based on criteria such as number of peptides 
 or spectra identified for a given protein .
we opted not to do so to 
 highlight the capabilities of psea - quant , which can identify putative 
 biologically significant protein sets in large noisy proteomics data 
 sets .
nevertheless , psea - quant might benefit from contaminant filtering 
 and other sophisticated data set postprocessing techniques .
contaminant 
 filtering may be particularly helpful before a psea - quant analysis 
 of proteomics experiments involving the use of antibodies such as 
 affinity purification coupled to mass spectrometry .
indeed , protein 
 nonspecific binding to antibodies could hinder the analytical power 
 of psea - quant .
there are several publically available computational 
 approaches that address this problem ( saint , comppass , or decontaminator ) that could be used upstream of psea - quant in 
 the context of affinity purification coupled to mass spectrometry 
 experiments . on the other hand , psea - quant may in some cases be helpful 
 to detect contaminants ( see the supporting information ) . among the significant protein sets outputted by psea - quant
. this 
 problem would be important if the list of significant protein sets 
 outputted by psea - quant would be very long and tedious to analyze . 

this does not appear to be the case in the context of our present 
 study . nevertheless ,
in the future , we will propose methods to merge 
 protein sets that share an important subset of proteins .
several methods 
 that have been proposed to measure the distance or the semantic similarity 
 between gene sets , such as the one from go , will be explored .
psea - quant p - value estimation accuracy is 
 limited 
 by the number of samplings performed by the monte carlo procedure . 
 to obtain an extremely high accuracy ,
a very large number of samplings 
 is required , which in turn necessitates a considerable computational 
 running time .
therefore , in the near future we will explore computational 
 strategies , such as genetic algorithms , to accelerate psea - quant and 
 provide an even more accurate p - value estimator .
this paper focused on finding protein sets that were significantly 
 enriched for proteins that are quantified with a low coefficient of 
 variation and have a high abundance .
however , one may not be interested 
 in protein sets involving abundant proteins , but rather protein sets 
 enriched for proteins with low abundance variance among a set of replicates . 

psea - quant can easily be adapted to compute the significance of such 
 protein sets by modifying the enrichment score weight matrix to only 
 account for quantification reproducibility .
furthermore , the enrichment 
 score weight matrix can be modified to consider additional features 
 besides abundance and abundance coefficient of variation , such as 
 the number of unique peptides detected or spectra intensities .
the ability to identify protein sets enriched for proteins with 
 high abundance and reproduced quantification yields useful information 
 about the proteome of a given sample .
we firmly believe that methods 
 such as psea - quant will significantly improve our understanding of 
 large quantification proteomics data sets .
it will also provide helpful 
 guidance to help formulate hypotheses about the biological mechanisms 
 and processes taking place in a given sample .
 the majority of large - scale 
 proteomics quantification methods yield 
 long lists of quantified proteins that are often difficult to interpret 
 and poorly reproduced . 
 computational approaches are required to analyze 
 such intricate quantitative proteomics data sets . 
 we propose a statistical 
 approach to computationally identify protein sets ( e.g. , gene ontology 
 ( go ) terms ) that are significantly enriched with abundant proteins 
 with reproducible quantification measurements across a set of replicates . 
 to this end , we developed psea - quant , a protein set enrichment analysis 
 algorithm for label - free and label - based protein quantification data 
 sets . 
 it offers an alternative approach to classic go analyses , models 
 protein annotation biases , and allows the analysis of samples originating 
 from a single condition , unlike analogous approaches such as gsea 
 and psea . 
 we demonstrate that psea - quant produces results complementary 
 to go analyses . 
 we also show that psea - quant provides valuable information 
 about the biological processes involved in cystic fibrosis using label - free 
 protein quantification of a cell line expressing a cftr mutant . 
 finally , 
 psea - quant highlights the differences in the mechanisms taking place 
 in the human , rat , and mouse brain frontal cortices based on tandem 
 mass tag quantification . 
 our approach , which is available online , 
 will thus improve the analysis of proteomics quantification data sets 
 by providing meaningful biological insights . 
Introduction
Methods
Results
Discussion and Conclusion