the clinical proteomic 
 tumor analysis consortium ( cptac ) was charged 
 with establishing proteome characterization centers to measure protein 
 differences across large numbers of tumors from the cancer genome 
 atlas ( tcga ) . to ensure that the differences 
 they found were reproducible , each center conducted many replicate 
 analyses for a pair of patient - derived xenograft tissues as a comparative 
 reference material (  compref  ) ,
first as a preliminary 
 validation of their workflows and later interspersed among the tcga 
 samples to ensure ongoing validation of their workflows .
ms / ms experiments , represent a unique 
 opportunity to evaluate the variability of protein differentiation 
 technologies across multiple platforms .
variability can enter 
 proteomic experiments through sample handling 
 and proteolysis , prefractionation , liquid chromatography , mass spectrometry 
 configuration , and bioinformatics . at the start of the program in 2011 , cptac evaluated 
 ischemia time in the context of global proteomes and phosphotyrosine signaling to evaluate potential impacts for proteomic changes after blood 
 supply to a tissue
is lost but before the sample is frozen . at the 
 same time , cptac conducted platform validation for each proteome characterization 
 center by distributing aliquots of the compref xenografts ; data from 
 each site could then be used to assess the variability contributed 
 by analytical methodology .
most sites opted to modify their platforms 
 in response to these data , most commonly by switching to brplc fractionation from scx or ief fractionation techniques .
patient - derived tumor xenografts are 
 a technology by which tumors 
 can be induced in mice via cells taken from human tumors , affording 
 better latitude for experimentation . when 
 tissue is harvested from a xenograft
, it will contain both human and 
 mouse proteins since the tumor cells are from a different species 
 than the host environment .
the xenografts employed in this study were 
 whim2 ( basal ) and whim16 ( luminal - b ) , drawn from a larger breast cancer 
 study at washington university in st .
both xenografts produced large tumors ; they were grown in a sufficient 
 number of mice to generate a pool of protein that was large enough 
 to serve all of the cptac proteome characterization centers . while 
 the differences between basal and luminal - b tumors have been studied 
 at length , the proteomic differences between them can not be scored 
 against a comprehensive answer key .
these experiments with whim2 
 and whim16 represent the three chief 
 techniques used for differentiating proteins in discovery proteomics 
 experiments .
itraq is one of the most common isobaric labeling strategies 
 in use at present ; after digestion with 
 trypsin , peptides from each sample are labeled with different chemicals 
 of nearly identical mass on their n - termini and lysine side chains , 
 and then peptides from different samples can be mixed together for 
 lc  ms / ms analysis .
ions of a particular peptide from multiple 
 samples will be isolated and fragmented together , each producing a 
 characteristic reporter ion at low m / z in the ms / ms scan .
the intensities of these reporter ions reflect 
 the quantity of that peptide in the source samples .
spectral counting 
 is a label - free strategy by which the number of spectra attributed 
 to a particular protein from an individual sample can be compared 
 to the number of spectra observed for that protein in another sample 
 of similar complexity .
alternatively , 
 data of this type may be analyzed by integrating the extracted ion 
 chromatograms for intact peptide ions drawn from successive ms scans . a protein that matches to intense ions in the 
 first experiment but less intense ions in a second experiment
both analyses of 
 label - free data , however , rely on the ability to control for higher 
 overall identification rates in one sample than in another . measuring the reproducibility of differential proteomics
a team from surromed evaluated the stability 
 of peptide intensities measured from many replicates of a reference 
 plasma sample in lc 
this work was extended to encompass isotopic 
 labeling strategies with an ab sciex qstar xl by kim et al . in 2007 .
old et al . found that spectral counting methods 
 detected differences better and that peptide intensities estimated 
 enrichment ratios better from thermofinnigan lcq deca data using scx 
 or gel exclusion fractionation in 2005 .
a similar 2005 investigation by zybailov et al . found that spectral 
 counting methods and isotope enrichment methods produced highly correlated 
 results , but spectral counting was more reproducible across their 
 replicates . in 2006 and 2007 , the phillip 
 c. wright laboratory published two investigations into the reproducibility 
 of itraq quantitation , producing a coefficient of variation of 0.09 
 and probing sources of variations in itraq reporter ion intensities 
 and documenting the value of replicates in these studies .
patel et al . applied itraq labeling and label - free mass spectrometry - based 
 proteomics approaches to the proteome of the bacterium methylocella silvestris , and the results showed good 
 agreement between the itraq experiment and the label - free approach 
 for relative quantification . in 2012 , 
 wang et al . reported a comparative study of itraq and label - free lc - based 
 quantitative proteomics approaches using two chlamydomonas 
 reinhardtii strains
the 
 comparison indicated that the label - free method provided better quantitation 
 accuracy for high fold change ratios and identified 40% more proteins , 
 but the itraq method showed better quantitation accuracy and reproducibility . 
 in a 2013 study of human adenovirus infections by trinh et al .
, a label - free method showed higher levels of 
 protein up- or downregulation in comparison to itraq - labeled samples , 
 and data suggested that the label - free method was more accurate than 
 the itraq method .
compared label - free and label - based 
 strategies for proteome analysis of hepatoma cell lines , showing that the label - free approach outperformed 
 tmt methods regarding proteome coverage , but the label - free method 
 was found to be less accurate than tmt approaches .
in addition to 
 these comparative studies , the literature is also rich with papers 
 on the advantages and disadvantages of different strategies of quantitative 
 mass spectrometry in proteomics . in general
, these 
 technology assessments have been limited in the 
 number of replicates and instruments they include , and the time frame 
 from first to last replicate has been relatively short .
the cptac 
 compref experiments , on the other hand , span six different instruments 
 and range up to 10 months in duration .
because the same pair of pooled 
 samples is represented in each experiment , the data are informative 
 for technical replicate variation rather than biological replicate 
 variation .
the data supporting this study 
 were generally drawn from interstitial experiments ; the whim2 and 
 whim16 samples were run as periodic quality controls in a longer series 
 of tcga samples .
this study evaluates these data to answer the following 
 key questions:are differential proteomics experiments 
 repeatable for a particular workflow?are difference lists reproducible 
 in comparing two different workflows?do the various differential proteomics 
 technologies see the same biology ?
are differential proteomics experiments 
 repeatable for a particular workflow ? are difference lists reproducible 
 in comparing two different workflows ?
patient - derived xenograft 
 tumors from 
 established basal ( whim2 ) and luminal - b ( whim16 ) breast cancer intrinsic 
 subtypes were raised subcutaneously in 8 week old 
 nod.cg-prkdc il2rg / szj mice ( jackson 
 laboratories , bar harbor , me ) as previously described .
have significantly different gene expression and proteomic 
 signatures that are related to their 
 intrinsic biology and endocrine signaling . tumors from each animal 
 were harvested by surgical excision at approximately 1.5 cm with minimal ischemia time by immediate immersion in a liquid nitrogen 
 bath .
the tumor tissues were then placed in precooled tubes on dry 
 ice and stored at 80 c .
a tissue pool of cryopulverized 
 tumors was prepared in order to generate sufficient material that 
 could be reliably shared and analyzed among multiple laboratories .
briefly , tumor pieces were transferred into precooled covaris tissue - tube 
 1 extra ( tt01xt ) bags ( covaris no .
520007 ) and processed in a covaris 
 cryoprep cp02 device using different impact settings according to 
 the total tumor tissue weight : < 250 mg = 3 ; 250350 mg = 
 4 ; 350440 mg = 5 ; and 440550 mg = 6 .
1131 - 436 ) on 
 dry ice , and the tissue was thoroughly mixed with a metal spatula 
 precooled in liquid nitrogen .
the tissue powder was then partitioned 
 ( 100 mg aliquots ) into precooled cryovials ( corning no .
( note that cryopulverized tissue will melt if it is transferred to 
 a plastic weighing boat . )
all procedures were carried out on dry ice 
 to maintain tissue in a powdered , frozen state .
each site processed 
 the tissue powder by independent protocols ; protein denaturation and 
 digestion were not controlled by a cptac - wide standard operating procedure .
ms / ms experiments from the whim2 and whim16 samples ; 
 thermo fisher ltq orbitrap velos , orbitrap elite , and q - exactive models 
 were included .
three instruments ( ovelos@10 , ovelos@45 , and qexac@56 , 
 where the number represents an anonymized cptac institution ) produced 
 itraq 4plex experiments . in each 4plex , 
 two channels represented each of the two samples .
three other instruments 
 ( ovelos@65 , oelite@65 , and qexac@98 ) produced label - free experiments 
 in which whim2 and whim16 were analyzed separately . all sites except 
 site 65 employed hcd for data - dependent ms / ms production , with orbitrap 
 measurement of fragment ions . both ovelos@65 and oelite@65
the 
 resulting tables are available as a microsoft excel spreadsheet in 
 the supporting information . instrument - specific 
 details are provided in supporting information method a. to maximize information yield 
 from these data , three different search engines were applied to each 
 lc  ms / ms experiment ( figure 1 ) .
myrimatch 2.1.138 , ms - gf+ 
 versions 9630 and 9979 , and comet version 
 2014.01 searched a fasta sequence database 
 containing ncbi refseq human ( 32  799 sequences , downloaded 
 sept 7 , 2011 ) , ncbi refseq mouse ( 29  617 sequences , downloaded 
 march 4 , 2011 ) , and the porcine trypsin sequence .
each sequence was 
 reversed in silico by the search engines for target - decoy estimation 
 of error rates ; the decoy sequences from all three search engines 
 were denoted by an xxx prefix .
fully tryptic and semitryptic peptides 
 were allowable matches . in all cases , a precursor mass tolerance of 
 20 ppm was applied , allowing for an error of one neutron in monoisotope 
 selection .
site 65 data were measured in the ion trap , so fragment 
 ions were allowed to vary by up to 0.5 m / z in myrimatch or within a one m / z bin by comet and ms - gf+ . for all other sites ,
fragments 
 were required to fall within 20 ppm of expected m / z by myrimatch , within a 0.01 m / z bin by comet , or within the hcd model of ms - gf+ . 

post - translational modifications expected a mass shift of 57.021464 
 da on all cys , and all algorithms allowed met to gain 15.994915 da 
 through oxidation as a dynamic modification .
ms - gf+ also allowed for 
 acetylation of n - termini ( + 42.010565 da ) . up to two missed cleavages 

a mass shift of 144.102063 
 da was assumed to be found at both n - termini and lys residues ( and 
 the itraq protocol was applied in ms - gf+ ) .
comet and myrimatch results 
 were exported to pepxml files , whereas 
 ms - gf+ results were written to mzidentml format .
label - free 
 sets were processed once by a spectral counts method and once by extracted 
 ion chromatograms .
while instrument - specific assemblies of psms were 
 used for repeatability analysis , an all - instrument assembly was analyzed 
 for reproducibility and biological pathway and network enrichment .
identification scores were translated 
 to q - values in idpicker 
 3.1 , build 599 .
the psm fdr ( false discovery rate ) was limited to 0.5% , 
 with the fdr being estimated by doubling the number of decoy hits 
 and dividing by the total passing a threshold .
the number of spectra 
 required per protein was then increased from two until the empirical 
 protein fdr had fallen below 5% .
for each site , the number of spectra 
 required per protein was as follows : ovelos@10 = 7 , ovelos@45 = 3 , 
 qexac@56 = 19 , ovelos@65 = 4 , oelite@65 = 2 , and qexac@98 = 4 .
idpicker 
 assembled protein groups using parsimony rules that eliminated subset 
 and subsumable proteins from the list and grouped indiscernible proteins .
idpicker matched each protein name from refseq 
 to an hgnc ( hugo gene nomenclature committee ) gene symbol for human 
 or to a mgi ( mouse gene informatics ) gene symbol for mouse .
if neither 
 was available for a protein name , then the software used the name 
 of the protein preceded by unmatched .
the software allows for graphical 
 or script - driven extraction of quantities in the form of spectral 
 counts ( spc ) or intensities , organized for this study by gene rather 
 than protein groups .
if spectra could be attributed to multiple transcripts 
 of a particular gene , then reporting by gene would consolidate all 
 of the psms to the single gene source , thus reducing the number of 
 hypotheses to be tested in finding expression differences .
two kinds 
 of intensities can be used : itraq reporter ion values can be summed 
 across all psms for each channel for each gene or extracted precursor 
 ion chromatograms ( xic ) in label - free experiments can be integrated 
 and summed across all psms for each gene . in itraq , idpicker computes sums rather than ratios , though users 
 can subsequently compute ratios from the summed intensity for each 
 gene .
this places a greater weight on genes with intense psms than 
 on those with weak signals .
when inferring xic intensities , the software 
 was not directed to integrate precursor intensity in the absence of 
 a confident psm through retention time mapping due to the large numbers 
 of lc 
the software exported 
 these quantitative tables by the command line idpquery tool from the 
 idpicker sqlite databases . for cross - site and biological pathway 
 and network analysis , a conjoint 
 assembly of data from all sites
ms / ms 
 experiments , the conjoint assembly spanned 3285 pepxml / mzidentml files . 
 as in the individual instrument assemblies , a psm fdr of 0.5% was 
 applied . requiring 20 spectra per gene pushed the empirical protein 
 fdr below 5% .
the 
 raw abundance data , 
 either spectral counts or itraq and xic intensities data , need to 
 be normalized before any statistical analysis .
variation in sampling 
 handling and sample loading across experiments can lead to variation 
 in the gene abundance and thus introduce systematic biases into the 
 differentiation study . to correct for the overall experiment - wise 
 difference , the spectral counts were normalized by the overall abundance 
 of each experiment using the total spectral counts in an experiment .
itraq and xic intensities data were normalized 
 by the median intensities in an experiment .
though 
 the qspec method was originally employed 
 in label - free proteomics data , the modeling framework can be similarly 
 applied to model the reporter ion intensities from itraq data . to 
 compare the log ratios between whim2 and whim16 , a linear mixed effect 
 regression model
was constructed for xic and itraq intensities and 
 a generalized linear mixed effect model for spectral counts .
a detailed 
 description of the method is presented in supporting information method b. fortran source code for this differentiation 
 is available at http://homepages.uc.edu/~wang2x7/research.htm .
the estimation of gene variance was a key factor differentiating 
 the per - instrument and cross - instrument analyses ( figure 1 ) .
in one - versus - one replicate 
 comparison , the normal distribution was assumed for log xic and itraq 
 intensities .
all of the nondifferential genes were assumed to have 
 the same variance , and the constant variance was inferred from the 
 pool of nondifferential genes detected by the model .
this was a strong 
 assumption that the variances are the same for all intensity levels , 
 but a check on the fitted model confirmed that the equal variance 
 assumption is approximately satisfied ( see supporting information figure s1 ) . for spectral count data ,
the poisson 
 distribution was assumed , and thus the variance was set to equal the 
 mean .
a check on the data showed that overdispersion may have affected 
 the data , leading to an underestimate of variance .
a negative binomial 
 model was fitted , and its results were very similar to those under 
 a poisson assumption .
by contrast , in the multiple replicates comparison 
 used for the cross - instrument reproducibility evaluation , gene variances 
 were estimated individually for both the xic / itraq intensities and 
 the spectral count cases .
for all genes observed from the xenograft 
 samples , only genes for 
 which the geneid contained at least one human gene name were considered . 

, we set up a requirement of minimum evidence for 
 a gene to be included in statistical differentiation analysis .
following 
 bing zhang s minimal spectral count requirement , we required a minimum of 1.4 spectra , on average , 
 for multiple replicate comparison . for example , in ovelos@65 ( a  j ) , 
 28 spectra were needed before a gene was evaluated for differentiation 
 ( 1.4  10 replicates  2 samples ) . in one - versus - one comparison , 
 we set the threshold as 2 spectra per replicate .
additionally , 
 handling zero values for spectral counts , xics , or 
 itraq intensities deserves special consideration as a zero may be 
 interpreted as either a missing value ( nas ) or a measured zero .
genes 
 that had only zero spectral counts in either whim2 or whim16 were 
 excluded ( this may eliminate some number of genuinely infinite differences , 
 though most will be low - information genes ) .
these spectral count requirements 
 were applied prior to itraq or xic differentiation as well .
for genes 
 with a mixture of zero and positive values in abundance , the handling 
 of spectra counts data differs from that of itraq or xic data . in 
 modeling of spectral counts ,
for the continuous itraq and 
 xic intensity data , we noticed that zeroes mostly likely came from 
 sources other than biological reasons , such as intensities below the 
 detection limit or failure in peak detection . treating zeros
the same 
 as positive values in the modeling may severely violate the log - normal 
 distribution assumptions on the continuous data .
we thus treated zeroes 
 in itraq and xic as missing values ; they were not used in differentiation 
 analysis . the difference in modeling zero values
may lead to contradictory 
 differentiation results from using spectral count data as noted in 
 the results and discussions ; one potential 
 solution is to model both zero and positive parts in differentiation 
 analysis using zero - inflated models .
determining 
 the intersection of differential genes in the context of biology emphasized 
 a subset of the overall data .
the analysis included these data sets : 
 ovelos@10 ( itraq , 6 replicates ) , ovelos@45 ( itraq , 5 replicates ) , 
 qexac@56 ( itraq , 5 replicates ) , ovelos@65 ( spc , first 10 replicates ) , 
 ovelos@65 ( xic , first 10 replicates ) , qexac@98 ( spc , first 4 replicates ) , 
 and qexac@98 ( xic first 4 replicates ) .
the oelite@65 data were excluded 
 because the instrument was known to be in deficient operating order 
 at the time the 4 replicates were collected .
the last 6 replicates 
 of ovelos@65 were excluded because they were interstitial with a different 
 type of sample than the first 10 replicates .
the final 6 replicates 
 of qexac@98 were excluded because ion transmission through the quadrupole 
 was considerably lower during data collection . to perform pathway 
 enrichment , we eliminated genes that were mapped only to the mouse 
 genome . when both human and mouse gene names were associated with 
 spectra , only the human name was retained . in cases
where expression 
 could be attributed to multiple isoforms , we chose to keep the first 
 isoform listed , and when a natural read - through event was indicated 
 ( two gene symbols separated by a hyphen ) , only the first symbol was 
 retained .
though these heuristic filters could conceivably introduce 
 biases , we note that these rules were applied only to 20 multiple 
 isoform proteins and 6 read - through proteins of the 8126 genes considered 
 ( 0.25% ) .
fisher s exact test was used to examine the statistical enrichment 
 of those 
 genes called as more highly expressed in whim2 or in whim16 for each 
 of the 188 pathways relative to all those genes not in the specified 
 pathway .
the genes upregulated for each of the two xenografts were 
 handled separately to avoid losing statistical power .
hochberg procedure , and we considered multiple - test - corrected p values 
 below 0.05 to be significant .
we also interpreted the genes 
 expressed significantly highly in 
 whim2 or in whim16 against the context of a global protein 
we first identified the hierarchical modules 
 from the iref network based on the netsam algorithm .
then , based on fisher s exact test , we calculated 
 the p values of enrichment for all modules in each 
 differentially expressed gene set identified by the seven data sets . 

hochberg 
 procedure . applying an fdr limit of 5% , we finally identified whim2- 
 or whim16-specific enriched modules for each of seven data sets .
all 
 of the above procedures were implemented in the netgestalt web tool 
 ( http://www.netgestalt.org ) .
we also used netgestalt to 
 covisualize the enriched modules of seven data sets for identification 
 of the functional consistency .
the 
 six instruments included in this study varied considerably in labeling 
 and fractionation strategy , lc gradients and mass spectrometry instruments , 
 and replicate schedule and data replacement policy . correspondingly , 
 large differences appeared in rates of ms / ms collection .
the two q - exactive 
 instruments in the study produced high maximum sustained rates of 
 ms / ms collection : 9.10 hz for qexac@56 and 9.28 hz for qexac@98 , exceeding 
 the best rate of acquisition in orbitraps for ms / ms scans ( 4.35 hz 
 for cid in ovelos@65 or 3.89 hz for hcd in ovelos@45 ) .
translating 
 these spectra to identified psms produced a less clear distinction 
 based on instrument type ( figure 2 ) .
while ovelos@65 produced relatively low sensitivity 
 in terms of distinct peptides , averaging 58  028 identified 
 spectra ( 26  250 distinct peptide sequences ) from its 15 fractions 
 per sample , it showed an encouraging degree of consistency across 
 10 months of continuous operation .
qexac@98 used essentially the same 
 amount of time per sample , though split to only five fractions , and 
 its first four experiments for the two samples identified an average 
 of 99  916 spectra ( 53  045 distinct peptide sequences ) ; 
 unfortunately , this rate tapered off considerably , and by the time 
 replicates six through ten were collected , identification rates were 
 much lower .
the instrument operator reported that the ion transmission 
 through the quadrupole worsened significantly in later runs .
viewing 
 these results instead in terms of discernible protein groups flattened 
 the differences among instruments .
the cumulative counts of distinct 
 proteins at < 5% empirical protein fdr were ovelos@10 = 11  428 , 
 ovelos@45 = 11  603 , qexac@56 = 15  655 , ovelos@65 = 8640 , 
 oelite@65 = 4199 , and qexac@98 = 10  435 .
note that these counts 
 included both mouse and human proteins since a xenograft incorporates 
 proteins from both species .
when 
 data from each instrument were assembled separately with 0.5% psm 
 fdr and < 5% empirical protein fdr , identification sensitivity varied 
 considerably by site .
ailing instrument oelite@65 yielded the lowest 
 sensitivity by far , whereas qexac@56 produced a remarkable number 
 of identifications .
label - free instruments ran whim2 and whim16 separately , 
 whereas itraq instruments combined these samples into a single 4plex . rather than emphasize distinguishable 
 protein groups , we projected 
 psms to the genes from which those peptides derived .
this projection 
 ensured that spectra associated with peptides shared among multiple 
 isoforms transcribed from the same gene would be counted only once .
in addition , reporting spectra per gene makes 
 it simple to relate proteomics data to biological pathways and networks 
 that are described by the use of hgnc identifiers . when multiple genes 
 represented exactly the same sets of peptides , they were reported 
 as an indiscernible gene group .
repeatability of identification 
 scaled with the numbers of spectra 
 identified for each gene group .
first , the genes were separated into exponentially sized bins : 47 , 
 815 , 16 - 31 , 3263 , 64127 , 128255 , 256511 , 
 and 5121023 spectra across all replicates of both whim2 and 
 whim16 .
this excluded 585 proteins with disproportionately high spectral 
 counts , such as the maximum for mouse albumin : 63  734 .
an average 
 of 998 genes fell into each bin , with the highest spectral count bin 
 including 610 genes . for each bin , we computed the average number 
 of replicates in which the genes produced any identifiable spectra 
 ( out of 16 replicates ) . for whim2 ,
the averages were 2.57 , 4.96 , 8.29 , 
 11.71 , 13.98 , 15.18 , 15.58 , and 15.83 .
whim16 produced very similar 
 averages : 2.12 , 3.96 , 7.00 , 10.37 , 13.23 , 15.12 , 15.72 , and 15.93 . 

of the 8566 genes ovelos@65 identified , 3616 were identified universally 
 in whim2 and 3306 were identified universally in whim16 .
a total of 
 2730 genes were identified universally across both whim2 and whim16 . 
 when a similar analysis was performed for the itraq experiments of 
 ovelos@10 , the bins began at 7 spectra because that was the minimum 
 number required to control the protein fdr .
even in the set of genes 
 with only 713 spectra in evidence , the identification repeatability 
 rate was quite high : 5.12 of 6 replicates .
once genes had produced 
 28 or more spectra in aggregate , the average identification repeatability 
 was close to 6.00 .
of the 11  284 genes that ovelos@10 identified , 
 9822 were identified in all six replicates .
reports from idpicker 
 were used to find two sets of differential 
 genes from each label - free replicate , one based on spectral counts 
 ( spc ) and another based on extracted ion chromatograms ( xics ) and 
 to find one set of differential genes from each itraq replicate ( see 
 experimental procedures : statistical differentiation ) .
this part of the study sought to determine the reproducibility 
 of differentiation for individual experiments rather than across multiple 
 replicates . for label - free experiments , this would mean comparing 
 the spectral counts or precursor intensities from a set of fractions 
 representing one analysis of whim2 to the comparable data from a set 
 of fractions representing one analysis of whim16 ( analyses comparing 
 multiple replicates are performed below ) . in these itraq experiments , 

each experiment inherently produced duplicate information for whim2 
 and for whim16 ; this analysis , however , compares data from a single 
 channel of whim2 to a single channel of whim16 . to determine 
 differences , we chose strategies that have been evaluated 
 in the peer - reviewed literature .
we opted to employ a pooled variance 
 model for the single - replicate comparisons ; similar strategies have 
 been implemented as local - pooled - error , empirical bayes , and other tests for 
 microarray experiments .
the approach embodied in idpicker for extracting 
 precursor ion intensity has been published separately , including an evaluation of the rate of false 
 attribution of intensity to a peptide . because the volume of data 
 was so extensive , we opted not to take advantage of the ability to 
 align retention time for fractions across replicates . as a result , 

a peptide that was present in both replicates contributed a spectrum 
 count or precursor intensity when it was identified in a replicate 
 but contributed no count or intensity when it was not identified in 
 a replicate . in all analyses ,
evidence is summed at the level of the 
 gene that gave rise to proteoforms to 
 which counts or intensity were identified . working with ratios
is 
 certainly more common in itraq analysis , but idpicker 3.1 was capable 
 of intensity output only , not ratios , summing the precursor intensities 
 for all psms supporting the gene identification . submitted work by 
 jian - ying zhou demonstrated that this method is equivalently resistant 
 to false positives while producing high data set correlations .
, who concluded that  our results demonstrate that 
 the sum of intensities and total least squares algorithms provided 
 the most accurate estimates of protein abundance for a wide range 
 of simulated and experimental conditions .
the commonly used average 
 of ratios algorithm consistently provided estimates with the highest 
 errors .  the fraction of observed genes found to be differential 
 at a 5% 
 fdr varied strongly by instrument .
the itraq instruments yielded low 
 average differential gene fractions : ovelos@10 = 1.6% , ovelos@45 = 
 3.4% , and qexac@56 = 2.5% ; our investigation showed that the bayesian 
 model expected a high degree of variance in each gene s reporter 
 ion intensities when no biological difference was present ( the model 
 assumed that at least half of genes would not differ between the two 
 xenografts ) .
label - free data sets were analyzed twice : once solely 
 on the basis of spectral counts ( spc ) and once solely on the basis 
 of extracted ion current ( xic ) from precursor ions in ms scans .
xic 
 analysis found somewhat higher fractions of genes to be differential : 
 ovelos@65 = 5.0% , oelite@65 = 3.5% , and qexac@98 = 7.7% .
qexac@98 
 produced a wide range of differential gene fractions , reflecting the 
 drop in identification sensitivity ( figure 2 ) ; the first four replicates averaged 12.9% 
 differential genes , whereas the last six averaged 4.3% . shifting to 
 spc analysis instead ( which shifts to a poisson expectation of variance 
 in the statistical model )
altered the differential gene fractions : 
 ovelos@65 = 9.8% , oelite@65 = 3.5% , and qexac@98 = 5.3% ( 8.2% for 
 the first four replicates , and 3.4% for the last six ) .
we note that 
 the fraction of genes found to be differential in the ovelos@65 set 
 was larger under spc analysis , whereas the fraction was higher in 
 qexac@98 under xic analysis ; this may reflect the optimizations in 
 methodology applied at each site .
two different strategies 
 were applied to determine the repeatability of differential gene detection 
 among experiments from a given instrument .
the first simply asked 
 which differences are consistent with those from at least one other 
 replicate experiment on the same instrument ; these overlaps were also 
 evaluated through cohen s kappa statistic .
the second compared 
 the rankings of genes on an axis from highly upregulated in whim2 
 to unchanging to highly upregulated in whim16 .
both analyses compared 
 data from a single itraq channel to those from another single itraq 
 channel , despite the collection of duplicate data in itraq 4plexes . 

this decision was intended to reflect the realities of large - scale 
 itraq experiments , where duplicate data collection is rare .
figure 3 visualizes 
 the set of differential genes from each replicate on each instrument . 

differential genes were called common and colored blue if they were 
 also found as differences in other experiments from the same instrument . 

they were labeled unique and colored orange if they were found in 
 only one replicate from this instrument .
all four possible comparisons 
 were generated from the duplicate itraq experiments ; both the low 
 and high m / z channels for whim2 
 were compared to both the low and high m / z channels for whim16 .
agreement in itraq experiments was 
 defined somewhat more stringently than in label - free experiments to 
 reflect the use of this technology in experiments spanning many 4plexes . 

consider the case where whim2 was presented in duplicate on channels 
 114 and 115 , while whim16 was reported on the 116 and 117 channels . 

a comparison between channels 114 and 116 has mutual information with 
 a comparison between channels 114 and 117 .
even a comparison of 114/116 
 versus 115/117 , however , will have shared information because they 
 would be based on the same set of psms . as a result
, common differential 
 genes from itraq experiments were required to be confirmed by altogether 
 separate lc 
how many of 
 the differential genes from each experiment are confirmed 
 by at least one other replicate experiment ?
blue differential genes 
 are found in common by another experiment in this instrument , whereas 
 orange ones are unique to a single replicate . for itraq ,
note that instruments that produced more replicates were likely to 
 have a higher proportion of common differential genes by random chance .
the extent to which figure 3 is dominated by common differential 
 genes is an encouraging 
 development for this field ; the average percentage of differences 
 in common for each instrument was as follows : ovelos@10 = 88% , ovelos@45 
 = 81% , qexac@56 = 87% , ovelos@65-xic = 78% , oelite@65-xic = 61% , qexac@98-xic 
 = 63% , ovelos@65-spc = 93% , oelite@65-spc = 88% , and qexac@98-spc 
 = 91% .
the ailing oelite@65 generated a large proportion of unique 
 differences , but the overall numbers of differential genes were quite 
 low , and only three other replicates could confirm any particular 
 difference .
qexac@98 also presented a fairly large number of unique 
 differential genes , but it seems clear that the falloff in sensitivity 
 for the final six replicates was costly .
the high degree of agreement 
 seen in the latter replicates for this instrument ( e  j ) was 
 reassuring in that while reduced sensitivity led to shorter lists 
 of differences it did not compromise reproducibility . a more 
 restrictive analysis , based on cohen s kappa , compared 
 the genes declared to be differential in pairs of experiments drawn 
 from each instrument .
essentially , this is built around a contingency 
 table ; how many genes are differential in a and differential in b , 
 how many genes are differential in neither a nor b , and how many genes 
 are found to be differential in one but not the other . in computing 
 this metric , we included only those genes that appeared in both experiments . 

the itraq instruments produced median kappas in 
 the range of 0.49 to 0.57 , with maximum values universally higher 
 than 0.88 resulting from comparisons with a channel in common .
for 
 all three label - free instruments , the kappa values were substantially 
 higher for spc analysis than for xic analysis ( a difference of 0.18 
 to 0.34 ) , with median kappas of 0.50 for qexac@98-spc , 0.58 for ovelos@65-spc , 
 and 0.74 for oelite@65-spc .
signed log posterior probabilities 
 were produced for each gene 
 in each replicate , with the magnitude of the negative log posterior 
 probabilities indicating the significance with which a gene may be 
 called differential and the sign marking the orientation of that potential 
 difference .
the ranks of genes on these lists can then be compared 
 in a spearman rank correlation .
figure 4 shows four example plots of correlation coefficients 
 for signed log posterior probabilities from each replicate against 
 the values from each other replicate .
the average number of genes 
 included in each instrument for differentiation was as follows : ovelos@10 
 = 6571 , ovelos@45 = 5877 , qexac@56 = 8880 , ovelos@65-spc = 2381 , oelite@65-spc 
 = 1344 , and qexac@98-spc = 2643 ( xic counts were only slightly different ) . rank correlations 
 compare the ordering of genes by signed posterior 
 probabilities .
ovelos@10 illustrates the extent to which differential 
 probabilities within an itraq 4plex are more similar than those across 
 multiple 4plexes .
the qexac@56 shows an exception to this behavior 
 in replicate b. spectral count - based differentiation produces similar 
 overall correlations to itraq , though without the benefit of being 
 able to compare within lc  ms / ms experiments .
the declining sensitivity of identification 
 in qexac@98 gave low correlation values other than the one between 
 the first two replicates .
data from ovelos@10 differentiate the correlation values 
 within 
 a 4plex from those between different 4plex replicates . within a 4plex
correlation coefficients 
 between 4plexes , however , never rose as high as 0.8 but rarely fell 
 below 0.7 .
this disparity may result in large part from the fact that 
 intensities within a 4plex are drawn from exactly the same set of 
 psms for a given gene , rendering them more comparable than a case 
 where intensities are drawn from entirely different psms ( and likely 
 distinct sets of peptide sequences ) as disparate sets of lc 
qexac@56 generally follows the same pattern 
 of behavior , with an even greater difference between internal - to-4plex 
 and external comparisons . the behavior for replicate b , however , bears 
 closer attention .
the data from channels 114 and 115 were considerably 
 different , and the external correlations for channel 114 were lower 
 against all other replicates .
the result is a reminder that a great 
 many different factors must all go well for reproducible proteomic 
 differentiation experiments .
( note , however , that the internal data 
 analysis pipeline by site 56 did not show a reduced correlation for 
 this channel .
the internal pipeline used a different search engine , 
 a peptide ratio - based assessment , and a pearson correlation on log 
 fold change . )
label - free experiments may be simpler to conduct 
 at the bench , 
 but they are also subject to variation .
the 16 replicates from ovelos@65 
 in figure 4 reveal 
 a general decrease in correlation for replicate g. looking back to figure 3 reveals an increased 
 proportion of unique differential genes for this replicate ; replicate 
 g also produced the lowest kappa metric for this instrument in both 
 spc and xic analysis .
qexac@98 was subject to considerable variation in identification 
 sensitivity ( figure 2 ) , and its correlation values were consistently lower than those 
 for other instruments .
all of the correlation plots 
 can be found in supporting information figure s3 . comparing across instruments
made it possible 
 to take advantage 
 of the replicates produced for each instrument .
instead of building 
 differential gene lists from individual instrument assemblies , the 
 cross - instrument evaluation was developed from an assembly that spanned 
 all of the data ( figure 1 and experimental procedures : bioinformatics 
 and biostatistics : protein identification ) . at an empirical 
 protein
fdr of 4.58% , the assembly included 12  741 protein 
 groups supported by a total of 6  980  499 psms .
the bayesian 
 model was able to estimate variance per gene through the use of replicates , 
 improving its discrimination power .
the number of 
 gene groups tested for differentiation in each instrument was as follows : 
 ovelos@10 = 6570 , ovelos@45 = 5744 , and qexac@56 = 8081 .
single - replicate 
 analysis had shown a maximum of 3.4% differential genes for these 
 three instruments , but those percentages rose significantly when the 
 software was able to estimate variance using replicate measurements 
 for each gene : ovelos@10 = 31.7% , ovelos@45 = 9.4% , and qexac@56 = 
 39.3% . when the first three and last three replicates for ovelos@10 
 were considered separately , the number of differential genes plummeted 
 to 13.3 and 14.2% for the early and late sets , respectively ; doubling 
 from three to six replicates greatly improved the statistical power 
 for detecting differential genes .
ovelos@45 , by contrast , identified 
 the fewest overall genes and yielded the least differentiation among 
 the itraq instruments . examining the underlying data revealed higher 
 typical variance per gene than that for other sites , and the operators 
 of this instrument noted that each replicate was a full process replicate 
 ( repeating digestion , labeling , and mixing ) rather than a technical 
 replicate that simply repeated lc 
an internal analysis by this site showed good consistency 
 in the genes found to be differential for each replicate ( see supporting 
 information figure s4 ) .
the label - free 
 data from ovelos@65 and qexac@98 provided enough 
 replicates to allow for subset comparisons , whereas oelite@65 lagged 
 enough in performance that it was not examined in detail .
the first 
 10 and last 6 replicates from ovelos@65 were split to two sets , reflecting 
 that the former were interstitial with tcga samples and the latter 
 were interstitial with normal tissue .
similarly , the data from qexac@98 
 for the first 4 replicates were separated from the last 6 replicates 
 to reflect the different schedules for collecting these two sets . 

p = 3219 , oelite@65 = 1896 , qexac@98a  d 
 = 4893 , and qexac@98e  j = 2758 .
the counts for label - free instruments 
 were taken from the spc analysis ; the number of genes compared by 
 xic was slightly lower since some precursor ions were not matched . 

the differential gene fractions by spc analysis were 64.2 and 48.7% 
 for the early and late sets for ovelos@65 , whereas the qexac@98 percentages 
 were 34.1 and 37.0% ( note , though , that the later set from this instrument 
 included substantially fewer genes ) .
analyzing the same data by xic , 
 though , yielded 51.9 and 59.6% differential genes for ovelos@65 , demonstrating 
 that having more replicates is not a guarantee for a higher fraction 
 of differential genes . the xic analysis for qexac@98 produced 32.3% 
 differential genes in the high - sensitivity
early set but 39.8% in 
 the later , less sensitive runs for this instrument . since spc 
 and xic analyses are generated from the same underlying 
 psms , one might expect that their lists of differences are highly 
 similar .
however , the sum of precursor ion intensity observed for 
 a given gene may be stable while spectral counts vary between two 
 cohorts , or vice versa .
difference lists from three data sets were 
 interrogated for this purpose : ovelos@65a  j , ovelos@65k  p , 
 and qexac@98a  d .
oliveros ) compared the lists of up - in - whim16 and up - in - whim2 genes 
 for both xic and spc analysis ( see supporting information figure s5 ) . of all genes named as up - in - whim2 
 in these three sets , from 47 to 64%
of 
 all genes named as up - in - whim16 in the three sets , from 53 to 75% 
 were named in both analyses .
the two techniques contradicted each 
 other in only one category : genes named as up - in - whim2 by spc but 
 up - in - whim16 by xic .
the first 10 replicates of ovelos@65 produced 
 two genes in this conflict , while the last 6 replicates produced another 
 9 .
examining underlying data from 
 these conflicts yielded examples of three phenomena . in the first 
 case , the numbers of identified spectra and the sums of precursor 
 intensity simply point in opposite directions . in the second case , 

zero counts were treated as observations in spc analysis but as missing 
 data in xic analysis , placing greater weight on replicates that reported 
 nonzero intensities . in the third case , sorting genes by log2 ratios for fdr determination introduced noise that could be avoided 
 by sorting on regression coefficients instead .
given that the number 
 of differential genes in a list ranged from 928 to 1328 , these conflicts 
 never rose above 1% , which we judged to be tolerable in this study .
venn analysis of differential genes from the three itraq instruments 
 shows that almost every difference found by ovelos@45 was also found 
 by the other two instruments ( see supporting information figure s6 for these cross - instrument images ) . 
 among the 1403 up - in - whim2 genes ,
the bulk of the gene differences was found by qexac@56 only 
 ( 58% ) , by ovelos@10 only ( 18% ) , or by both instruments ( 18% ) .
a similar 
 pattern appears in the 2269 up - in - whim16 genes , with 33% found by 
 qexac@56 only , 9% by ovelos@10 only , 38% found by both these instruments , 
 and 19% found by all three instruments .
the data for label - free instruments 
 showed great consistency between early and late sets of ovelos@65 
 data along with a substantial number of differences found only by 
 qexac@98 ( presumably due to its high sensitivity ) . whereas more spc 
 differences were observed by ovelos@65a 
j , the xic analysis 
 favored the later ovelos@65k  p set instead . figure 5 visualizes 
 the extent of agreement for differential genes found in one set of 
 replicates versus those found in others . whether analyzing the data 
 by spc or by xic , many of the differences found in the first four 
 replicates of qexac@98 were not reproduced by other label - free sets 
 ( perhaps because of the increased sensitivity achieved in these experiments ) . 

a similar phenomenon arose from qexac@56 ; this instrument produced 
 by far the highest psm sensitivity of the study , and the many additional 
 differential genes inferred from its data could not be confirmed by 
 other data sets .
the low number of overall differences found in the 
 ovelos@45 set reflects a relatively low overall gene sensitivity and 
 a low degree of differential discrimination ; by contrast , the differences 
 found by this instrument were almost entirely consistent with the 
 differences produced by others .
to what extent are the differential genes 
 found for each data set 
 confirmed by other data sets ?
those represented by blue were found 
 to be differential in common with another data set in that graph panel , 
 as well .
orange genes , on the other hand , were unique to a particular 
 data set .
the fraction 
 of differential genes confirmed by another instrument in the fourth 
 panel of figure 5 ranged 
 from 67% for ovelos@65a  j by spc to 99% for ovelos@45 .
however , 
 biological insight is often derived by projecting data at the level 
 of the functional pathway or subnetwork . to examine the stability 
 of biological insight that could be derived from data generated on 
 each platform , we calculated the statistical enrichment of genes identified 
 as more highly expressed in whim2 or in whim16 in each kegg pathway 
 and each network module generated from the iref protein 
protein 
 interaction network ( see experimental procedures ) using fisher s exact test with benjamini 
overall , many more proteins 
 were identified as more highly expressed in whim16 than were identified 
 as more highly expressed in whim2 .
four kegg pathways ( antigen processing 
 and presentation , arginine / proline metabolism , glutathione metabolism , 
 and glycolysis / gluconeogenesis ) were enriched in up - in - whim16 lists 
 from all platforms .
the top 10 most significant pathways from each 
 platform showed consistency , though these sets varied ( table 1 ) .
focal adhesion was ranked 
 as first or second most significant by the three itraq data sets , 
 and it was in the top 10 most significant pathways for both partitions 
 of the ovelos@65 data under both spc and xic methods , but it was not 
 determined to be significant by qexac@98 by either analysis . similarly , 
 five network modules were identified as the top 10 most significant 
 modules enriched with up - in - whim16 lists from at least four platforms 
 ( table 2 and figure 6 ) .
only two modules 
 were enriched for the qexac@98 data under the spc method , and no module 
 was enriched for the qexac@98 data under the xic method ( figure 6 ) .
specially , modules 
 level_2_module_3 ( response to wounding ) and level_2_module_11 ( immune 
 system processes ) were ranked as first or second most significant 
 modules by four platforms .
the up - in - whim2 genes were less consistent 
 in both pathway and network analyses , but this was largely due to 
 the fact that far fewer genes were observed to be significantly differential 
 in whim2 by any platform , and pathway enrichment overall was much 
 lower .
 indicates that the pathway 
 was in the top 10 most significant pathways for that data set .
+ indicates 
 that the pathway was significant ( corrected p value 
 < 0.05 ) in that data set .
enriched function is the most 
 significant go ( gene ontology ) biological process term enriched with 
 genes in the module .
 indicates that the pathway was in the 
 top 10 most significant pathways for that data set .
netgestalt - based analysis of functional consistency 
 for up - in - whim16 
 genes from seven data sets .
( a ) proteins in the iref network are placed 
 in a linear order together with the hierarchical modular organization 
 of the network .
( b  g ) for each data set , the network modules 
 found to be enriched are colored light red . within these modules , 

we also evaluated the possibility of permutation - based 
 fdr estimation 
 for pathway enrichment at a 10% threshold . under these circumstances , 

qexac@98 resembled the other instruments to a greater degree , with 
 significant hits occurring in an additional 10 pathways appearing 
 in table 1 by spc analysis . 

the permutation analysis also found 5 additional pathways to be significant 
 in table 1 by xic analysis . 
 at this time
, netgestalt has not yet gained the ability to perform 
 permutation - based fdr correction .
differential 
 proteomic technologies are complex , and that complexity 
 opens the door to many potential sources of variability . just as proteomic 
 inventory experiments sample from a large potential pool of peptides , 
 differential proteomics experiments sample from a large potential 
 pool of differential proteins / genes . given that the underlying inventories 
 from these six instruments ranged more than an order of magnitude 
 in the number of distinct peptides they identified ( figure 2 ) , the degree of conformity 
 in the differential gene lists across instruments was rather surprising 
 ( figure 5 ) .
projecting 
 into kegg pathways or iref biological networks further reinforced 
 this agreement among instruments .
quality control methods and 
 standard operating procedures are a 
 necessary part of clinical proteomics , and sharing results for common 
 samples between laboratories can be a powerful means to evaluate performance 
 in both proteomic inventories and differentiation . the association 
 of biomolecular resource facilities has been particularly influential 
 in such cross - laboratory studies . when qc methods highlight an lc 
ms / ms 
 experiment from a fractionated set as deficient , however , the ideal 
 response is to rerun the entire fractionated set rather than replacing 
 a subset of the fractions . to replace only the problematic fractions 
 can increase the variability of the set by adding in the drift of 
 the instrument as a function of time .
figure 4 illustrates 
 an important principle for isobaric labeling studies : intensity values 
 within an itraq 4plex replicate were more comparable than values between 
 different sets of replicates because they are drawn from the same 
 set of psms .
the speed and capacity of these methods clearly exceed 
 those of label - free approaches as a result of multiplexing .
current 
 isobaric reagents can support analysis of the greater than 100 sample 
 comparisons carried out by the cptac data production centers through 
 the use of a common pooled sample included in each 4plex or through 
 a randomized block design .
the degree of multiplexing possible through 
 use of current isobaric labeling reagents is at 10plex through use 
 of the tandem mass tag ( tmt , thermo ) reagents , and further increases 
 in the degree of multiplexing are anticipated as new chemistries and 
 encoding strategies are developed . to return to the questions that motivated this study
, it appears 
 clear that repeatable performance in differential proteomics is feasible , 
 whether isobaric labels or label - free methods are employed .
the repeatability 
 of ovelos@65 over 10 months of nearly continuous data acquisition 
 was quite high , though the sixth of 16 replicates was somewhat questionable 
 in the correlation analysis of figure 4 .
the diminished performance in the final six replicates 
 of qexac@98 is a clear reminder that instrument performance needs 
 to be comparable for differentiation to be repeatable .
the large increase 
 in differential genes for the cross - instrument reproducibility analysis 
 reminds us of the value of multiple replicates for differential proteomics . 

while this study did not take advantage of full biological replicates 
 for its two samples , these are certainly valuable in most experiments . 
 using the information from technical replicates for this study
greatly 
 increased the number of differential genes by enabling per - gene variability 
 estimates .
technical variation remains a fact of life for all proteomics 
 laboratories ; this study , like many others in the field , might have 
 benefited from the use of randomization and blocking designs in addition 
 to its use of replication .
if laboratories 
 deploy different methodologies to analyze the differences 
 between the same two complex samples , then they will assuredly see 
 differences in the gene or protein lists produced by the two technologies . 

when label - free data were analyzed by spectral counting rather than 
 precursor intensity , the differences yielded a high degree of overlap . 

when itraq rather than label - free methods are deployed , the differential 
 genes were again quite similar .
these overlaps suggest a degree of 
 maturity in proteomic methods that has grown through years of development 
 along multiple tracks . at base , biologists need to know that 
 differential proteomics technologies 
 can produce meaningful results .
our assessment showed that biological 
 pathway and network analysis is highly consistent across instruments . 

which subset of genes is observed to be differentially regulated may 
 be sampled from a larger set associated with a pathway or network . 

the pattern of these differences yields a stronger signal in combination . 
 with these questions addressed ,
we hope to see wider deployment of 
 differential proteomics in support of clinical and biological studies .
 the nci clinical proteomic tumor 
 analysis consortium ( cptac ) employed 
 a pair of reference xenograft proteomes for initial platform validation 
 and ongoing quality control of its data collection for the cancer 
 genome atlas ( tcga ) tumors . 
 these two xenografts , representing basal 
 and luminal - b human breast cancer , were fractionated and analyzed 
 on six mass spectrometers in a total of 46 replicates divided between 
 itraq and label - free technologies , spanning a total of 1095 lc  
 ms / ms 
 experiments . 
 these data represent a unique opportunity to evaluate 
 the stability of proteomic differentiation by mass spectrometry over 
 many months of time for individual instruments or across instruments 
 running dissimilar workflows . 
 we evaluated itraq reporter ions , label - free 
 spectral counts , and label - free extracted ion chromatograms as strategies 
 for data interpretation ( source code is available from http://homepages.uc.edu/~wang2x7/research.htm ) . from these assessments 
 , we found that differential genes from 
 a single replicate were confirmed by other replicates on the same 
 instrument from 61 to 93% of the time . 
 when comparing across different 
 instruments and quantitative technologies , using multiple replicates , 
 differential genes were reproduced by other data sets from 67 to 99% 
 of the time . 
 projecting gene differences to biological pathways and 
 networks increased the degree of similarity . 
 these overlaps send an 
 encouraging message about the maturity of technologies for proteomic 
 differentiation . 
Introduction
Experimental
Procedures
Results and Discussion
Conclusions