Answers.Body: <p>The GATK team maintains a GitHub repository with their pipelines: <a href="https://github.com/gatk-workflows" rel="nofollow noreferrer">https://github.com/gatk-workflows</a>. They use the <a href="https://cromwell.readthedocs.io/en/stable/" rel="nofollow noreferrer">Cromwell Workflow Management System</a>, an open source tool created by the Broad Institute.</p>
  <p>You are probably most interested in <a href="https://github.com/gatk-workflows/gatk4-data-processing" rel="nofollow noreferrer">https://github.com/gatk-workflows/gatk4-data-processing</a>, their workflow for aliging fastq files against a reference genome to produce a bam file which can be used for downstream analyses such as variant calling:</p>
  <blockquote>
  <h1>gatk4-data-processing</h1>
  <h2>Purpose :</h2>
  <p>Workflows for processing high-throughput sequencing data for variant discovery with GATK4 and related tools.</p>
  <h2>processing-for-variant-discovery-gatk4 :</h2>
  <p>The processing-for-variant-discovery-gatk4 WDL pipeline implements data
  pre-processing according to the GATK Best Practices. The workflow
  takes as input an unmapped BAM list file (text file containing paths
  to unmapped bam files) to perform preprocessing tasks such as mapping,
  marking duplicates, and base recalibration. It produces a single BAM
  file and its index suitable for variant discovery analysis using tools
  such as Haplotypecaller.</p>
  <ul>
  <li>If you are starting with FASTQ files visit the seq-format-conversion repository for workflows to convert FASTQs to unmapped BAMS.</li>
  <li>The processing-for-variant-discovery-gatk4 provides quick and general processing for sequence data using the latest releases of
  GATK. If users are interested in a more elaborate version of this
  workflow with quality control tasks and routinely tested for validity
  (useful in production environments) then visit the
  gatk4-genome-processing-pipeline repository.</li>
  <li>The BAM output from processing-for-variant-discovery-gatk4 can be used to perform a variety of other analysis like somatic short variant
  discovery, germline short variant discovery, or germline copy number
  variant discovery. Visit the GATK Best Practices documentation to
  determine what to do next with the BAM files.</li>
  </ul>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T09:55:36.763Z
Answers.LastActivityDate: 2023-04-03T09:55:36.763Z
Answers.UserDisplayName: terdon
Answers.Body: <p>I think the nextflow core (<a href="https://nf-co.re" rel="nofollow noreferrer">https://nf-co.re</a>) pipelines should satisfy most of your requirements. These pipelines are written in nextflow (<a href="https://nextflow.io" rel="nofollow noreferrer">https://nextflow.io</a>) which allows scaling from personal to hpc infrastructure rather efficiently.
  The core pipelines are usually quite well documented, and have a module design. I have not really looked into the testing methodology, but I think the modules themselves are tested. Running them is easy once nextflow is installed on your system.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T08:38:44.923Z
Answers.LastActivityDate: 2023-04-03T08:38:44.923Z
Answers.UserDisplayName: Niklas
Body: <p>I am a software engineer and I am preparing a presentation to aspiring bioinformatics PhDs on how to use best-practice software engineering when publishing code (such as include documentation, modular design, include tests, ...). In particular my presentation will be focused on publishing &quot;pipeline&quot; code, that is, code that is mainly focused on transforming data to a suitable shape for analysis.</p>
  <p>I am trying to find good example of published bioinformatics pipelines that I can point students to, but as I am not a bioinformatician I am struggling to do so, so would like your help. It doesn't matter if the published pipeline is super-niche or not very popular so long as you think it is engineered well so that I can look it up.</p>
  <p>Specifically the published code should have: adequate documentation, testing methodology, modular design and be easy to install.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-03T06:30:07.670Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-03T07:42:27.740Z
Comments.Text: @Community I added the list of features that the published package should have to the end of the question.
Comments.UserDisplayName: nsimplex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-04T15:44:29.697Z
Comments.Text: One quite important point IMO, how are you publishing, e.g. GitHub, Pypy, Docker, Conda? Publishing a data pipeline is not the same as placing monolithic code on GitHub that entirely belongs to the author and has no dependencies subject restricted copyright.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-04T19:56:05.593Z
Comments.Text: This question is too broad to be useful for the StackExchange format (which works best if there's a single, specific correct answer). I recommend that you ask this at a place where more general discussion is easier, like [reddit](https://reddit.com/r/bioinformatics).
Comments.UserDisplayName: gringer
Tags: <data-preprocessing><data-retrieval><publishing>
Title: Where can I find examples of published bioinformatics pipeline packages?
ThreadText: # Where can I find examples of published bioinformatics pipeline packages?
  
  (Asked by: username_0 on 2023-04-03)
  
  I am a software engineer and I am preparing a presentation to aspiring bioinformatics PhDs on how to use best-practice software engineering when publishing code (such as include documentation, modular design, include tests, ...). In particular my presentation will be focused on publishing "pipeline" code, that is, code that is mainly focused on transforming data to a suitable shape for analysis.
  
  I am trying to find good example of published bioinformatics pipelines that I can point students to, but as I am not a bioinformatician I am struggling to do so, so would like your help. It doesn't matter if the published pipeline is super-niche or not very popular so long as you think it is engineered well so that I can look it up.
  
  Specifically the published code should have: adequate documentation, testing methodology, modular design and be easy to install.
  
  **Tags:** data-preprocessing, data-retrieval, publishing
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  > **username_0**: @username_1 I added the list of features that the published package should have to the end of the question.  
  > **username_2**: One quite important point IMO, how are you publishing, e.g. GitHub, Pypy, Docker, Conda? Publishing a data pipeline is not the same as placing monolithic code on GitHub that entirely belongs to the author and has no dependencies subject restricted copyright.  
  > **username_3**: This question is too broad to be useful for the StackExchange format (which works best if there's a single, specific correct answer). I recommend that you ask this at a place where more general discussion is easier, like [reddit](https://reddit.com/r/bioinformatics).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2023-04-03. Score: 4
  
  The GATK team maintains a GitHub repository with their pipelines: <https://github.com/gatk-workflows>. They use the [Cromwell Workflow Management System](https://cromwell.readthedocs.io/en/stable/), an open source tool created by the Broad Institute.
  
  You are probably most interested in <https://github.com/gatk-workflows/gatk4-data-processing>, their workflow for aliging fastq files against a reference genome to produce a bam file which can be used for downstream analyses such as variant calling:
  
  > # gatk4-data-processing
  >
  > ## Purpose :
  >
  > Workflows for processing high-throughput sequencing data for variant discovery with GATK4 and related tools.
  >
  > ## processing-for-variant-discovery-gatk4 :
  >
  > The processing-for-variant-discovery-gatk4 WDL pipeline implements data
  > pre-processing according to the GATK Best Practices. The workflow
  > takes as input an unmapped BAM list file (text file containing paths
  > to unmapped bam files) to perform preprocessing tasks such as mapping,
  > marking duplicates, and base recalibration. It produces a single BAM
  > file and its index suitable for variant discovery analysis using tools
  > such as Haplotypecaller.
  >
  > * If you are starting with FASTQ files visit the seq-format-conversion repository for workflows to convert FASTQs to unmapped BAMS.
  > * The processing-for-variant-discovery-gatk4 provides quick and general processing for sequence data using the latest releases of
  >   GATK. If users are interested in a more elaborate version of this
  >   workflow with quality control tasks and routinely tested for validity
  >   (useful in production environments) then visit the
  >   gatk4-genome-processing-pipeline repository.
  > * The BAM output from processing-for-variant-discovery-gatk4 can be used to perform a variety of other analysis like somatic short variant
  >   discovery, germline short variant discovery, or germline copy number
  >   variant discovery. Visit the GATK Best Practices documentation to
  >   determine what to do next with the BAM files.
  
  
  ---
  
  ### Answer by username_5 on 2023-04-03. Score: 4
  
  I think the nextflow core (<https://nf-co.re>) pipelines should satisfy most of your requirements. These pipelines are written in nextflow (<https://nextflow.io>) which allows scaling from personal to hpc infrastructure rather efficiently.
  The core pipelines are usually quite well documented, and have a module design. I have not really looked into the testing methodology, but I think the modules themselves are tested. Running them is easy once nextflow is installed on your system.
  
  
  ---
  
  