Answers.Body: <p>Great question! Note that from a prescriptive standpoint, the terms <strong><em>pipeline</em></strong> and <strong><em>workflow</em></strong> don't have any strict or precise definitions. But it's still useful to take a descriptive standpoint and discuss how the terms are commonly used in the bioinformatics community.</p>
  
  <p>But before talking about pipelines and workflows, it's helpful to talk about programs and scripts. A <strong><em>program</em></strong> or <strong><em>script</em></strong> typically implements a single data analysis task (or set of related tasks). Some examples include the following.</p>
  
  <ul>
  <li><a href="https://www.bioinformatics.babraham.ac.uk/projects/fastqc/" rel="noreferrer">FastQC</a>, a program that checks NGS reads for common quality issues</li>
  <li><a href="http://www.usadellab.org/cms/?page=trimmomatic" rel="noreferrer">Trimmomatic</a>, a program for cleaning NGS reads</li>
  <li><a href="https://combine-lab.github.io/salmon/" rel="noreferrer">salmon</a>, a program for estimating transcript abundance from NGS reads</li>
  <li>a custom R script that uses <a href="https://bioconductor.org/packages/release/bioc/html/DESeq2.html" rel="noreferrer">DESeq2</a> to perform differential expression analysis</li>
  </ul>
  
  <p>A <strong><em>pipeline</em></strong> or a <strong><em>workflow</em></strong> refers to a particular kind of program or script that is intended primarily to <em>combine other independent programs or scripts</em>. For example, I might want to write an RNA-seq workflow that executes Trimmomatic, FastQC, salmon, and the R script using a single command. This is particularly useful if I have to run the same command many times, or if the commands take a long time to run. It's very inconvenient when you have to babysit your computer and wait for step 3 to finish so that you can launch step 4!</p>
  
  <p>So when does a program become a pipeline? Honestly, there are no strict rules. In some cases it's clear: the 10-line Python script I wrote to split Fasta files is definitely <em>NOT</em> a pipeline, but the 200-line Python script I wrote that does nothing but invoke 6 other bioinformatics programs definitely <em>IS</em> a pipeline. There are a lot of tools that fall in the middle: they may require running multiple steps in a certain order, or implement their own processing but also delegate processing to other tools. Usually nobody worries too much about whether it's "correct" to call a particular tool a pipeline.</p>
  
  <p>Finally, a <strong><em>workflow engine</em></strong> is the software used to actually execute your pipeline/workflow. As mentioned above, general-purpose scripting languages like Bash, Python, or Perl can be used to implement workflows. But there are other languages that are designed specifically for managing workflows. Perhaps the earliest and most popular of these is GNU Make, which was originally intended to help engineers coordinate software compilation but can be used for just about any workflow. More recently there has been a proliferation of tools intended to replace GNU Make for numerous languages in a variety of contexts. The most popular in bioinformatics seems to be <a href="https://snakemake.readthedocs.io/en/stable/" rel="noreferrer">Snakemake</a>, which provides a nice balance of simplicity (through shell commands), flexibility (through configuration), and power-user support (through Python scripting). Build scripts written for these tools (i.e., a <code>Makefile</code> or <code>Snakefile</code>) are often called pipelines or workflows, and the <strong><em>workflow engine</em></strong> is the software that executes the workflow.</p>
  
  <p>The workflow engines you listed above (such as argo) can certainly be used to coordinate bioinformatics workflows. Honestly though, these are aimed more at the broader tech industry: they involve not just workflow execution but also hardware and infrastructure coordination, and would require a level of engineering expertise/support not commonly available in a bioinformatics setting. This could change, however, as bioinformatics becomes more of a "big data" endeavor.</p>
  
  <p>As a final note, I'll mention a few more relevant technologies that I wasn't able to fit above.</p>
  
  <ul>
  <li><a href="https://www.docker.com/" rel="noreferrer">Docker</a>: managing a consistent software environment across multiple (potentially dozens or hundreds) of computers; Singularity is Docker's less popular step-sister</li>
  <li><a href="https://www.commonwl.org/" rel="noreferrer">Common Workflow Language (CWL)</a>: a generic language for declaring how each step of a workflow is executed, what inputs it needs, what outputs it creates, and approximately what resources (RAM, storage, CPU threads, etc.) are required to run it; designed to write workflows that can be run on a variety of workflow engines</li>
  <li><a href="https://dockstore.org/" rel="noreferrer">Dockstore</a>: a registry of bioinformatics workflows (heavy emphasis on genomics) that includes a Docker container and a CWL specification for each workflow</li>
  <li><a href="https://toil.readthedocs.io/en/latest/" rel="noreferrer">toil</a>: a production-grade workflow engine used primarily for bioinformatics workflows</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-29T14:55:12.040Z
Answers.Text: Thanks, Daniel. Your answer outlines very well the details about the bioinformatics workflows. Your argument, that a very high level of software engineering is needed to use my listed workflow engines, is on point . If I understand correctly generally speaking bioinformatics pipelines can be pressed into workflow engines. So thanks again. That helped a lot.
Answers.UserDisplayName: A.Dumas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-29T15:18:23.743Z
Answers.Text: Nice answer! I'm biased, but I feel like [nextflow](https://www.nextflow.io/) is also worth a mention :) And also [WDL](https://software.broadinstitute.org/wdl/) to complete the picture of common workflow tools. There's a nice "awesome-pipeline" repo list of workflow tools here too: https://github.com/pditommaso/awesome-pipeline - a little more comprehensive than the one in the original post, perhaps with more of a bioinformatics slant.
Answers.UserDisplayName: ewels
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T13:10:35.333Z
Answers.Text: Thanks for the comment @tallphil. I too am biased by the tools with which I have experience. :-) Feel free to edit my answer to add a bit about nextflow and WDL.
Answers.UserDisplayName: Daniel Standage
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-29T02:51:12.670Z
Answers.LastActivityDate: 2019-03-29T13:42:48.003Z
Answers.LastEditDate: 2019-03-29T13:42:48.003Z
Answers.UserDisplayName: Daniel Standage
Body: <p>I want to understand the difference between pipeline systems and workflow engines.</p>
  
  <p>After reading <a href="https://link.springer.com/article/10.1007/s41019-017-0047-z" rel="noreferrer">A Review of Scalable Bioinformatics Pipelines</a> I had a good overview of current bioinformatics pipelines. After some further research I found that there is collection of highly capable <a href="https://github.com/meirwah/awesome-workflow-engines" rel="noreferrer">workflow engines</a>. My question is then based on what I saw for <a href="https://github.com/argoproj/argo" rel="noreferrer">argo</a>. I would say I can be used as a bioinformatics pipeline as well. </p>
  
  <p>So how do bioinformatics pipelines differ from workflow engines?</p>
  
Tags: <software-usage><terminology><workflow-management>
Title: What is the difference between a Bioinformatics pipeline and workflow?
ThreadText: # What is the difference between a Bioinformatics pipeline and workflow?
  
  (Asked by: username_0 on 2019-03-28)
  
  I want to understand the difference between pipeline systems and workflow engines.
  
  After reading [A Review of Scalable Bioinformatics Pipelines](https://link.springer.com/article/10.1007/s41019-017-0047-z) I had a good overview of current bioinformatics pipelines. After some further research I found that there is collection of highly capable [workflow engines](https://github.com/meirwah/awesome-workflow-engines). My question is then based on what I saw for [argo](https://github.com/argoproj/argo). I would say I can be used as a bioinformatics pipeline as well.
  
  So how do bioinformatics pipelines differ from workflow engines?
  
  **Tags:** software-usage, terminology, workflow-management
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-29. Score: 18 | Accepted answer
  
  Great question! Note that from a prescriptive standpoint, the terms ***pipeline*** and ***workflow*** don't have any strict or precise definitions. But it's still useful to take a descriptive standpoint and discuss how the terms are commonly used in the bioinformatics community.
  
  But before talking about pipelines and workflows, it's helpful to talk about programs and scripts. A ***program*** or ***script*** typically implements a single data analysis task (or set of related tasks). Some examples include the following.
  
  * [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/), a program that checks NGS reads for common quality issues
  * [Trimmomatic](http://www.usadellab.org/cms/?page=trimmomatic), a program for cleaning NGS reads
  * [salmon](https://combine-lab.github.io/salmon/), a program for estimating transcript abundance from NGS reads
  * a custom R script that uses [DESeq2](https://bioconductor.org/packages/release/bioc/html/DESeq2.html) to perform differential expression analysis
  
  A ***pipeline*** or a ***workflow*** refers to a particular kind of program or script that is intended primarily to *combine other independent programs or scripts*. For example, I might want to write an RNA-seq workflow that executes Trimmomatic, FastQC, salmon, and the R script using a single command. This is particularly useful if I have to run the same command many times, or if the commands take a long time to run. It's very inconvenient when you have to babysit your computer and wait for step 3 to finish so that you can launch step 4!
  
  So when does a program become a pipeline? Honestly, there are no strict rules. In some cases it's clear: the 10-line Python script I wrote to split Fasta files is definitely *NOT* a pipeline, but the 200-line Python script I wrote that does nothing but invoke 6 other bioinformatics programs definitely *IS* a pipeline. There are a lot of tools that fall in the middle: they may require running multiple steps in a certain order, or implement their own processing but also delegate processing to other tools. Usually nobody worries too much about whether it's "correct" to call a particular tool a pipeline.
  
  Finally, a ***workflow engine*** is the software used to actually execute your pipeline/workflow. As mentioned above, general-purpose scripting languages like Bash, Python, or Perl can be used to implement workflows. But there are other languages that are designed specifically for managing workflows. Perhaps the earliest and most popular of these is GNU Make, which was originally intended to help engineers coordinate software compilation but can be used for just about any workflow. More recently there has been a proliferation of tools intended to replace GNU Make for numerous languages in a variety of contexts. The most popular in bioinformatics seems to be [Snakemake](https://snakemake.readthedocs.io/en/stable/), which provides a nice balance of simplicity (through shell commands), flexibility (through configuration), and power-user support (through Python scripting). Build scripts written for these tools (i.e., a `Makefile` or `Snakefile`) are often called pipelines or workflows, and the ***workflow engine*** is the software that executes the workflow.
  
  The workflow engines you listed above (such as argo) can certainly be used to coordinate bioinformatics workflows. Honestly though, these are aimed more at the broader tech industry: they involve not just workflow execution but also hardware and infrastructure coordination, and would require a level of engineering expertise/support not commonly available in a bioinformatics setting. This could change, however, as bioinformatics becomes more of a "big data" endeavor.
  
  As a final note, I'll mention a few more relevant technologies that I wasn't able to fit above.
  
  * [Docker](https://www.docker.com/): managing a consistent software environment across multiple (potentially dozens or hundreds) of computers; Singularity is Docker's less popular step-sister
  * [Common Workflow Language (CWL)](https://www.commonwl.org/): a generic language for declaring how each step of a workflow is executed, what inputs it needs, what outputs it creates, and approximately what resources (RAM, storage, CPU threads, etc.) are required to run it; designed to write workflows that can be run on a variety of workflow engines
  * [Dockstore](https://dockstore.org/): a registry of bioinformatics workflows (heavy emphasis on genomics) that includes a Docker container and a CWL specification for each workflow
  * [toil](https://toil.readthedocs.io/en/latest/): a production-grade workflow engine used primarily for bioinformatics workflows
  
  > **Comments:**
  > **username_0**: Thanks, username_1. Your answer outlines very well the details about the bioinformatics workflows. Your argument, that a very high level of software engineering is needed to use my listed workflow engines, is on point . If I understand correctly generally speaking bioinformatics pipelines can be pressed into workflow engines. So thanks again. That helped a lot.  
  > **username_2**: Nice answer! I'm biased, but I feel like [nextflow](https://www.nextflow.io/) is also worth a mention :) And also [WDL](https://software.broadinstitute.org/wdl/) to complete the picture of common workflow tools. There's a nice "awesome-pipeline" repo list of workflow tools here too: https://github.com/pditommaso/awesome-pipeline - a little more comprehensive than the one in the original post, perhaps with more of a bioinformatics slant.  
  > **username_1**: Thanks for the comment @tallphil. I too am biased by the tools with which I have experience. :-) Feel free to edit my answer to add a bit about nextflow and WDL.  
  
  ---
  
  