Answers.Body: <p>If your objective is to convert Bash scripts into Nextflow code, you might find Viash (<a href="https://viash.io" rel="noreferrer">https://viash.io</a>) a particularly useful open source tool. It automatically generates independent Nextflow modules from each of your scripts, along with the corresponding Docker containers for each module.
  Docker containers are generated on the fly based on your config description in Viash.
  These modules can be coupled together to build a (modular) Nextflow pipeline.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T06:58:17.493Z
Answers.Text: Thanks for answering the question as written! Super helpful!
Answers.UserDisplayName: Alex Reynolds
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-01T13:23:36.587Z
Answers.Text: Generating Docker containers on the fly is part of the behaviour of NextFlow *per se*, which is why it's so cool. However, I know zero about Viash. It would be useful if someone could give feedback on Viash.io because it may not possibly work as advertised here .... The question was flagged for hidden issues, i.e. it may not work as advertised, so I would recommend proceeding with care.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-30T07:10:57.053Z
Answers.LastActivityDate: 2023-05-30T07:10:57.053Z
Answers.UserDisplayName: Andy Boschmans
Answers.Body: <p>The gist would be something like:</p>
  <pre><code>fastqs = Channel.of(...some annoying groovy code to make this a list of tuples...)
  workflow{
    align(fastqs)
    makeGVCF(align.out.bams)
    ... collect() the output from above joint call them all...
  }
  
  process align {
    input:
      tuple val(sampleName), path(reads)
    output:
      tuple val(sampleName), path(&quot;${sampleName}.bam&quot;), emit: bams
    &quot;&quot;&quot;
    bwa ... | samtools sort ...
    &quot;&quot;&quot;
  }
  
  process makeGVCF{
    input:
      tuple val(sampleName), path(bam)
    output:
      path &quot;${sampleName}.gvcf&quot;, emit: gvcfs
    &quot;&quot;&quot;
    ...haplotype caller code...
    &quot;&quot;&quot;
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T01:11:28.297Z
Answers.Text: The usual way would be to use either the [`fromFilePairs`](https://www.nextflow.io/docs/latest/channel.html#fromfilepairs) factory method, or to read from a samplesheet using the [`fromPath`](https://www.nextflow.io/docs/latest/channel.html#frompath) factory and the [`splitCsv`](https://www.nextflow.io/docs/latest/operator.html#splitcsv) operator, e.g. `Channel.fromPath( params.samplesheet ).splitCsv( header:true ).map { row -> tuple( row.sample, [ row.fastq1, row.fastq2 ] ) }.set { readgroups }`. Then `align( readgroups )`.
Answers.UserDisplayName: Steve
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T05:41:00.000Z
Answers.Text: I really thank you both for your inputs. But my head started hurting when I reached the end of the comments. I guess I just have to grit my teeth and push through it.
Answers.UserDisplayName: Karthik Nair
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T17:14:06.207Z
Answers.LastActivityDate: 2023-05-03T17:14:06.207Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>Since you asked about templates, if your task is something rather generic, there is a good chance that a module, or several, that you can use to build your workflow exists in <a href="https://nf-co.re/modules" rel="nofollow noreferrer">https://nf-co.re/modules</a>
  Generally I would recommend taking a look at the source of nf-core pipelines (e.g. <a href="https://nf-co.re/sarek" rel="nofollow noreferrer">https://nf-co.re/sarek</a> for variant calling) to get an idea how to get your samplesheets in.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-05T20:24:41.680Z
Answers.Text: Nextflow (https://nextflow.io/) is a workflow manager / language used to implement analysis pipelines, nf-core is an open source collection of modules and pipelines written for / in nextflow
Answers.UserDisplayName: Niklas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-05T22:32:17.897Z
Answers.Text: Hi @Niklas thank you - I see your point. Its leveraging NextFlow and providing a link to a NextFlow template on GitHub. Got it. FYI I can use NextFlow. Its Docker container on the fly, is neat, but I agree with DevonRyan its Groovy (an obsolete language) I personally think is a bit weird.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-05T18:58:02.023Z
Answers.LastActivityDate: 2023-05-05T19:06:12.770Z
Answers.LastEditDate: 2023-05-05T19:06:12.770Z
Answers.UserDisplayName: Niklas
Body: <p>Not going to lie, my head hurts when I read the NextFlow documentation (No dig on the documentation, I think my head is losing its processing capacity lol).</p>
  <p>I had written a pipeline in Bash, essentially iterates over a text file containing full paths of R1 and R2 reads, followed by a series of commands and liberal use of gnu parallel to improve efficiency.</p>
  <p>Is there any template I can follow, where I can slot in my commands and get things running quickly?</p>
  <p>EDIT - The skeleton of pipeline goes as follows (As I'm not sure if I can put our inhouse pipeline in its entirety here) -</p>
  <p>Bash code which loops through R1 and R2, aligns with BWA-mem2 and sorts the bam, then using gnu parallel, I essentially follow the steps as outlined in GATK Best Practices (looking to drop BQSR, but the steps remain) to obtain a gvcf file. The steps in Best Practices have been turned to functions, rather than each step being called sequentially.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-03T14:36:14.470Z
Comments.Text: Please [edit] your question and show us your code. Nobody can help if they don't know what your code actually does. But, that said, every pipeline is different so I really doubt there will ever be a "template".
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-03T14:37:30.060Z
Comments.Text: Hi KarthikNair could you kindly supply the code or else a reproducible example? Otherwise there are a lot of NextFlow questions and answers on Bioinformatics SE, particularly answered by user @Steve, and those would likely provide pointers. To be honest it's not possible to answer the question as it stands in my personal opinion. I've a reasonable understanding of NextFlow, but I'm not an expert.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-03T15:10:51.993Z
Comments.Text: I have mentioned the brief gist about the pipeline, essentially we are following GATK best practices for variant calling (humans).
Comments.UserDisplayName: Karthik Nair
Tags: <bash><nextflow>
Title: Easier / lazy way to convert an existing bash pipeline to nextflow pipeline?
ThreadText: # Easier / lazy way to convert an existing bash pipeline to nextflow pipeline?
  
  (Asked by: username_0 on 2023-05-03)
  
  Not going to lie, my head hurts when I read the NextFlow documentation (No dig on the documentation, I think my head is losing its processing capacity lol).
  
  I had written a pipeline in Bash, essentially iterates over a text file containing full paths of R1 and R2 reads, followed by a series of commands and liberal use of gnu parallel to improve efficiency.
  
  Is there any template I can follow, where I can slot in my commands and get things running quickly?
  
  EDIT - The skeleton of pipeline goes as follows (As I'm not sure if I can put our inhouse pipeline in its entirety here) -
  
  Bash code which loops through R1 and R2, aligns with BWA-mem2 and sorts the bam, then using gnu parallel, I essentially follow the steps as outlined in GATK Best Practices (looking to drop BQSR, but the steps remain) to obtain a gvcf file. The steps in Best Practices have been turned to functions, rather than each step being called sequentially.
  
  **Tags:** bash, nextflow
  
  > **Comments:**
  > **username_1**: Please [edit] your question and show us your code. Nobody can help if they don't know what your code actually does. But, that said, every pipeline is different so I really doubt there will ever be a "template".  
  > **username_2**: Hi username_0 could you kindly supply the code or else a reproducible example? Otherwise there are a lot of NextFlow questions and answers on Bioinformatics SE, particularly answered by user @username_6, and those would likely provide pointers. To be honest it's not possible to answer the question as it stands in my personal opinion. I've a reasonable understanding of NextFlow, but I'm not an expert.  
  > **username_0**: I have mentioned the brief gist about the pipeline, essentially we are following GATK best practices for variant calling (humans).  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2023-05-30. Score: 5
  
  If your objective is to convert Bash scripts into Nextflow code, you might find Viash (<https://viash.io>) a particularly useful open source tool. It automatically generates independent Nextflow modules from each of your scripts, along with the corresponding Docker containers for each module.
  Docker containers are generated on the fly based on your config description in Viash.
  These modules can be coupled together to build a (modular) Nextflow pipeline.
  
  > **Comments:**
  > **username_4**: Thanks for answering the question as written! Super helpful!  
  > **username_2**: Generating Docker containers on the fly is part of the behaviour of NextFlow *per se*, which is why it's so cool. However, I know zero about Viash. It would be useful if someone could give feedback on Viash.io because it may not possibly work as advertised here .... The question was flagged for hidden issues, i.e. it may not work as advertised, so I would recommend proceeding with care.  
  
  ---
  
  ### Answer by username_5 on 2023-05-03. Score: 2
  
  The gist would be something like:
  
  ```
  fastqs = Channel.of(...some annoying groovy code to make this a list of tuples...)
  workflow{
    align(fastqs)
    makeGVCF(align.out.bams)
    ... collect() the output from above joint call them all...
  }
  
  process align {
    input:
      tuple val(sampleName), path(reads)
    output:
      tuple val(sampleName), path("${sampleName}.bam"), emit: bams
    """
    bwa ... | samtools sort ...
    """
  }
  
  process makeGVCF{
    input:
      tuple val(sampleName), path(bam)
    output:
      path "${sampleName}.gvcf", emit: gvcfs
    """
    ...haplotype caller code...
    """
  }
  
  ```
  
  > **Comments:**
  > **username_6**: The usual way would be to use either the [`fromFilePairs`](https://www.nextflow.io/docs/latest/channel.html#fromfilepairs) factory method, or to read from a samplesheet using the [`fromPath`](https://www.nextflow.io/docs/latest/channel.html#frompath) factory and the [`splitCsv`](https://www.nextflow.io/docs/latest/operator.html#splitcsv) operator, e.g. `Channel.fromPath( params.samplesheet ).splitCsv( header:true ).map { row -> tuple( row.sample, [ row.fastq1, row.fastq2 ] ) }.set { readgroups }`. Then `align( readgroups )`.  
  > **username_0**: I really thank you both for your inputs. But my head started hurting when I reached the end of the comments. I guess I just have to grit my teeth and push through it.  
  
  ---
  
  ### Answer by username_7 on 2023-05-05. Score: 2
  
  Since you asked about templates, if your task is something rather generic, there is a good chance that a module, or several, that you can use to build your workflow exists in <https://nf-co.re/modules>
  Generally I would recommend taking a look at the source of nf-core pipelines (e.g. <https://nf-co.re/sarek> for variant calling) to get an idea how to get your samplesheets in.
  
  > **Comments:**
  > **username_7**: Nextflow (https://nextflow.io/) is a workflow manager / language used to implement analysis pipelines, nf-core is an open source collection of modules and pipelines written for / in nextflow  
  > **username_2**: Hi @username_7 thank you - I see your point. Its leveraging NextFlow and providing a link to a NextFlow template on GitHub. Got it. FYI I can use NextFlow. Its Docker container on the fly, is neat, but I agree with username_5 its Groovy (an obsolete language) I personally think is a bit weird.  
  
  ---
  
  