the recent push for open data has also rapidly increased the availability of biomedical datasets for secondary analysis .
examples include the yale open data access project , a repository of clinical trial data , and the cancer genome atlas ( tcga ) , a project that makes genomic data accessible to researchers after initial findings are released .
while these data sets promote ongoing research , the ability to efficiently store , move , and analyze such large repositories is often a bottleneck to analysis .
in addition to the massive growth in volume and availability , novel analyses , including advanced statistical methods and machine learning , often require significant resources for efficient processing .
one example of this in biomedical research is the analysis of next generation sequencing ( ngs ) data .
ngs is also known as massively parallel or high - throughput sequencing , as it simultaneously sequences many fragments of dna , thereby producing enormous amounts of information .
in addition to being resource intensive , the reproducibility of computational experiments using these data is often limited due to the complexity of system and software configuration .
some application frameworks have made advances to improve the reproducibility of individual applications and analysis pipelines , but significant work remains to increase this reliability , particularly for experiments performed in resource - limited environments or on computational clusters .
, there has been significant progress in building virtualization layers for operating systems and more recently , software applications .
a current example of this includes the docker platform ( docker , san francisco , ca , usa ) , which allows for the creation and configuration of software containers for deployment on a range of systems .
while the use of these technologies has limitations , it also has the potential improve the usability of many software applications in computational biology .
as such , several studies and initiatives have begun to focus on the use of docker in bioinformatics and computer science research . in this paper , we demonstrate the potential benefits of containerized applications and application workflows for computational genomics research .
to augment an ongoing study related to tumor heterogeneity , we obtained access to acute myeloid leukemia ( aml ) ngs data from tcga .
aligned ngs data from tcga are available through the cancer genomics hub ( cghub ) .
the data set of interest consisted of approximately 12 terabytes ( tbs ) of whole genome sequencing ( wgs ) data and another 12 tb of whole exome sequencing data .
our analysis required the identification of somatic variants followed by a prediction of tumor heterogeneity using publicly available software tools .
unfortunately , many bioinformatics tools have specific software dependencies and natively run on only a subset of operating systems . in addition , many applications are unable to run their computations in parallel , thus limiting analysis throughput . while increasing the number of servers or individual server resources can improve analysis speed , overall processing may still be less efficient due to these limitations .
as previously noted , the docker platform allows for virtualized application deployments within a lightweight , linux - based wrapper called a container .
containerization enables developers to create virtual environments that have only the minimum necessary libraries , which users can quickly deploy on their own infrastructure in a secure , reproducible fashion .
in addition , the isolation offered by this approach means that a more robust , parallelized workflow can be created for some applications that do not natively support multi - threading or parallel processing . while this type of workflow implementation is not beneficial for all use cases , scenarios where compute capacity on a single node exceeds what a single application can utilize are likely to benefit from such an approach . to efficiently predict tumor heterogeneity from tcga data
, we implemented two key technologies : a python workflow using the luigi library ( spotify , stockholm , sweden ) and docker containers for each key application in our analysis pipeline .
while neither of these technologies is unique to genomic analysis or bioinformatics , luigi is a general workflow orchestration library , and docker allows for application visualization , as previously noted . for this use case , we were restricted to a single computational node with eight cores and 128 gigabyte ( gb ) of memory for analysis .
as shown in figure 1 , aligned wgs data were obtained for paired germline and tumor specimens from cghub ( now the genomic data commons ) using the cgdownload utility .
we identified somatic variants with the somaticsniper application and generated clonal predictions with the sciclone library .
sequential data acquisition and analysis took approximately 4 h to complete per paired sequence but had significant variability between specimens , since file size and the number of genetic mutations varied significantly .
serial workflow and architecture to download cancer genomics hub data . to obtain next generation sequencing data from cancer genomics hub ,
the cgdownload utility was used to transfer aligned whole genome and whole exome sequencing data .
the somaticsniper utility was then used to identify somatic variants and tumor clonality was predicted with sciclone .
because of the large number of specimens and the time needed for sequential analysis , our first approach to improving pipeline efficiency was the implementation of a python workflow using the luigi library .
this library is commonly used for workflow automation and has gained significant traction within bioinformatics through the development of the sci : luigi library .
implementation of our pipeline with this workflow library allowed us to include automated fault tolerance , primarily for issues that resulted in sequence download failures due to brief losses in network connectivity .
in addition , the workflow could run continuously and unsupervised , which markedly reduced the amount of hands - on time needed for analysis .
workflow automation led to a significant improvement in efficiency ; however , the inability to parallelize the software packages limited analysis throughput . while the cgdownload utility does support multi - threading , local bandwidth and disk size limitations made bulk downloading of the entire tcga data set difficult .
in addition , a bulk download followed by a full analysis of the data set would mean that network capacity would be saturated while the computational resources remained idle , followed by the opposite scenario . to maintain high , simultaneous utilization of all local hardware resources , including bandwidth , memory , and processor capacity
, we deployed the cgdownload utility , somaticsniper , and sciclone within isolated docker containers and executed each container with our luigi workflow [ figure 2 ] .
this approach allowed us to horizontally and vertically scale each application to take full advantage of our local hardware .
in addition , each application could be deployed on a single node running centos7 ( red hat , raleigh , nc , usa ) within a container running its natively - supported operating system : centos6 for cgdownload and ubuntu ( canonical , london , uk ) for somaticsniper and sciclone .
( a ) when deployed in a virtual server , the analysis workflow was installed on centos 6.7 and had to be run serially due to limitations in software parallelization and local resources .
applications are launched manually in sequence to download ngs data , identify variants , and predict tumor clonality .
( b ) when configured in docker containers and driven by a workflow manager , applications were automatically launched and able to scale based on available system resources .
each application was configured on its native operating system architecture within the container , as indicated in the figure this approach to application deployment can offer significant performance benefits .
however , any virtualization technology has the potential to offset these gains due to resource overhead in the virtualization layer . to assess the impact of docker virtualization on two key metrics , disk throughput , and processing efficiency , we used two benchmarking tools to evaluate performance on the virtual server as well as in a docker container . to benchmark disk input / output performance
, we used the dd command line tool , a standard linux utility that can be used to read and write files and to gather performance statistics .
the dd utility was used to write a 1 gb file and showed similar performance in both a virtual machine and a docker container within this same environment [ figure 3a ] .
similarly , results from sysbench , an open - source benchmarking utility originally created by mysql ab , found that the time needed to calculate 10,000 primes in either environment was equivalent [ figure 3b ] . when combined with evidence from other studies , these results demonstrate that docker has a minimal overhead for these components .
( a ) the time needed to write a one - gigabyte file with the dd utility was similar in both a virtual machine and within a docker container on the same host .
( b ) the calculation of 10,000 primes with the sysbench utility showed similar performance in a virtual machine and a docker container on the same host since we found similar benchmarking results for both the virtual machine and docker containers , we next executed our workflow to analyze the aml wgs results from tcga . using this approach
, we were able to stagger data download and processing to take advantage of all system resources [ figure 4 ] .
one limitation to this approach is that none of the tools described here provide built - in resource monitoring .
this pipeline was well - suited to parallel analysis since each application had specific , isolated resource needs such as network , storage , or compute capacity , which could be monitored with custom code within the python workflow .
use of this automated staggered workflow with docker containers allowed use to analyze fifty specimens from the tcga data set within approximately 3 days .
it is difficult to provide statistical performance metrics since many factors such as data volume , and network bandwidth can significantly alter the overall pipeline performance .
however , the general approach of staggered , parallel computation should provide increased processing efficiency for workloads such as this .
( a ) when performed serially , the download ( white bars ) and analysis ( shaded bars ) of a single pair of tumor and germline sequence on local hardware took approximately 4 h ( bars drawn to scale ) .
( b ) when parallelized with a workflow manager and docker containers , multiple specimens could be processed simultaneously to take advantage of all system resources , including network , memory , and processor capacity
the complex nature of genomic data and the tools used to analyze these data sets makes efficient processing difficult with standard environments . as noted above
, the use of emerging technologies such as docker in combination with automated workflows may significantly improve the efficiency of data processing in bioinformatics . with the growing number of open data projects , use of these techniques
will be necessary to take advantage of available computational resources . while performance and pipeline efficiency were key components of this implementation
, docker containers also allow for application isolation from the host operating system . since many bioinformatics tools have complex sets of dependencies and are difficult to build from source , the ability to deploy containers with different operating systems and dependency versions to the same host decreases the amount of effort needed to being analysis .
for example , the cgdownload utility is distributed as a compiled binary for use on centos 6.7 , but can only be deployed on centos 7 when built from source , which requires a significant amount of manual configuration . as shown in figure 2 ,
the use of containers allowed the deployment of each utility on its natively supported operating system , which improves stability and decreases the potential for dependency conflicts among software applications .
several other tools exist for the orchestration of containerized applications , such as kubernetes and docker swarm . for complex platforms , these tools can be used to deploy containers across hardware clusters and to integrate networking and storage resources between containers . however , these applications work strictly at the container level and do not inherently provide application - level workflows as presented here . additional implementation experience about the use of these tools within high - performance clusters may provide valuable insights about the scalability of these tools within bioinformatics workflows .
the above findings demonstrate the promise of emerging technologies to improve the efficiency of genomic analysis . because of the subsequent increase in analysis throughput , use of these tools means that big data analyses can be done even with limited local computational capacity .
finally , use of container technology can improve pipeline and experimental reproducibility since preconfigured applications can be readily deployed to nearly any host system .
while many factors can impact reproducibility , the use of containers limits variability due to differences in software environment or application configuration when appropriately deployed .
the continued use of emerging technology and novel approaches to software architecture has the potential to increase the efficiency of computational analysis in bioinformatics .
 background : the rapid acquisition of biological data and development of computationally intensive analyses has led to a need for novel approaches to software deployment . 
 in particular , the complexity of common analytic tools for genomics makes them difficult to deploy and decreases the reproducibility of computational experiments.methods:recent technologies that allow for application virtualization , such as docker , allow developers and bioinformaticians to isolate these applications and deploy secure , scalable platforms that have the potential to dramatically increase the efficiency of big data processing.results:while limitations exist , this study demonstrates a successful implementation of a pipeline with several discrete software applications for the analysis of next - generation sequencing ( ngs ) data.conclusions:with this approach , we significantly reduced the amount of time needed to perform clonal analysis from ngs data in acute myeloid leukemia . 
INTRODUCTION
TECHNICAL BACKGROUND
APPROACH
CONCLUSION
Financial Support and Sponsorship
Conflicts of Interest