Answers.Body: <p>Maybe take a look at workflow management systems, like</p>
  <ul>
  <li><a href="https://snakemake.readthedocs.io/en/stable/index.html" rel="nofollow noreferrer">snakemake</a></li>
  <li><a href="https://nextflow.io/example1.html" rel="nofollow noreferrer">Nextflow</a></li>
  <li><a href="https://toil.readthedocs.io/en/latest/" rel="nofollow noreferrer">Toil</a></li>
  <li><a href="https://cromwell.readthedocs.io/en/stable/" rel="nofollow noreferrer">Cromwell</a></li>
  </ul>
  <p>Admitted, each of those adds their additional learning curve. However, especially snakemake is not so much more complicated than a standard bash workflow. Those systems do help a lot organize and orchestrate your workflows</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-24T08:55:08.740Z
Answers.LastActivityDate: 2021-02-24T08:55:08.740Z
Answers.UserDisplayName: Carambakaracho
Answers.Body: <p>I understand the desire to pipe everything like that but I don't think it will work, even for such a small workflow. And as your pipe grows it'll make it harder to troubleshoot/debug.</p>
  <p>These days everyone uses workflow languages to write pipelines. It's a bit more work to get set up but worth it in the long run. I use WDL. Snakemake is very popular. BASH will of course work for small pipes but not with | for every step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-03T07:26:44.897Z
Answers.LastActivityDate: 2021-02-03T07:26:44.897Z
Answers.UserDisplayName: Liam McIntyre
Answers.Body: <p>Directly piping will definitely not work, as Liam McIntyre suggests. However, if you don't feel up to learning a workflow language just to do this, it is perfectly possible to just put the various commands in a bash driver script (or whatever you're comfortable with).</p>
  <p>Each of those tools will take parameters to define output filenames, or have systematic ways of writing FASTA files and defining inputs. You will have to look at the command line interface of each of those tools to figure it out (for example, sratools <a href="https://github.com/ncbi/sra-tools/wiki/08.-prefetch-and-fasterq-dump" rel="nofollow noreferrer">here</a>, bbduk.sh <a href="https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/bbduk-guide/" rel="nofollow noreferrer">here</a> and spades <a href="http://cab.spbu.ru/files/release3.12.0/manual.html" rel="nofollow noreferrer">here</a>). an example would be as follows:</p>
  <pre><code>#!/usr/bin/bash
  SRA=$1
  
  # get reads
  prefetch <span class="math-container">$SRA  # writes to a known path locally
  fasterq-dump $</span>SRA --split-3  # looks for path_to_sra_file and outputs it. assumes you are splitting.
  
  # trim reads
  bbduk.sh in1=<span class="math-container">$SRA''_1.fastq in2=$</span>SRA''_2.fastq out1=<span class="math-container">$SRA''_clean_1.fq out2=$</span>SRA''_clean_2.fq
  
  # re-QC the files- this step does not feed into the other steps!!
  fastqc <span class="math-container">$SRA''*.fastq $</span>SRA''*.fq  # runs fastqc on all fastq files
  
  # run assembly
  spades.py -o my_assembly --pe1 <span class="math-container">$SRA''_clean_1.fq --pe2 $</span>SRA''_clean_2.fq
  
  </code></pre>
  <p>and so on.</p>
  <p>I suppose that from a formal perspective it's better to use a workflow engine as Liam McIntyre says, but with such a straightforward workflow I'd say start simple and get more elaborate only if you need to; you'd spend 10x as long just figuring out the workflow language. This is ok if it's a goal to learn it or if you already know it, but if you just want some results putting it into a script is fine.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-17T22:15:10.810Z
Answers.Text: The other virtue of putting everything in a script is every command line option is documented.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-24T09:51:32.727Z
Answers.Text: All workflow management systems I have ever worked with also document all parameters (and much more) meticulously.
Answers.UserDisplayName: Pallie
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-17T21:52:05.920Z
Answers.LastActivityDate: 2021-02-17T21:52:05.920Z
Answers.UserDisplayName: Maximilian Press
Body: <p>I’m trying to create a pipeline for genome assembly. How best can I “redirect/pipe” from existing fasta files (or files in general) to other steps of the pipeline?</p>
  <p>I was thinking of going from the SRA Download step (prefetch and fasterq-dump) to fastqc, then to trimmomatic or bbduk to remove the adapters, one more fastqc check and then to SPAdes for the assembly.</p>
  <p>In my (amateur) mind it would be cool to be like:</p>
  <p>“SRRXXXXXX” | prefetch | fasterq-dump | fastqc | bbduk.sh | fastqc (second) | spades.py</p>
  <p>Any and all help is greatly appreciate! I know this stuff is pretty basic but I am so new to everything it’s painfully obvious to even me.</p>
  
Tags: <fasta><genome><assembly><sratoolkit><fastqc>
Title: Genome QC + Assembly Pipeline semantics
ThreadText: # Genome QC + Assembly Pipeline semantics
  
  (Asked by: username_0 on 2021-02-03)
  
  I’m trying to create a pipeline for genome assembly. How best can I “redirect/pipe” from existing fasta files (or files in general) to other steps of the pipeline?
  
  I was thinking of going from the SRA Download step (prefetch and fasterq-dump) to fastqc, then to trimmomatic or bbduk to remove the adapters, one more fastqc check and then to SPAdes for the assembly.
  
  In my (amateur) mind it would be cool to be like:
  
  “SRRXXXXXX” | prefetch | fasterq-dump | fastqc | bbduk.sh | fastqc (second) | spades.py
  
  Any and all help is greatly appreciate! I know this stuff is pretty basic but I am so new to everything it’s painfully obvious to even me.
  
  **Tags:** fasta, genome, assembly, sratoolkit, fastqc
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-02-24. Score: 1
  
  Maybe take a look at workflow management systems, like
  
  * [snakemake](https://snakemake.readthedocs.io/en/stable/index.html)
  * [Nextflow](https://nextflow.io/example1.html)
  * [Toil](https://toil.readthedocs.io/en/latest/)
  * [Cromwell](https://cromwell.readthedocs.io/en/stable/)
  
  Admitted, each of those adds their additional learning curve. However, especially snakemake is not so much more complicated than a standard bash workflow. Those systems do help a lot organize and orchestrate your workflows
  
  
  ---
  
  ### Answer by username_2 on 2021-02-03. Score: 0
  
  I understand the desire to pipe everything like that but I don't think it will work, even for such a small workflow. And as your pipe grows it'll make it harder to troubleshoot/debug.
  
  These days everyone uses workflow languages to write pipelines. It's a bit more work to get set up but worth it in the long run. I use WDL. Snakemake is very popular. BASH will of course work for small pipes but not with | for every step.
  
  
  ---
  
  ### Answer by username_3 on 2021-02-17. Score: 0
  
  Directly piping will definitely not work, as username_2 username_2 suggests. However, if you don't feel up to learning a workflow language just to do this, it is perfectly possible to just put the various commands in a bash driver script (or whatever you're comfortable with).
  
  Each of those tools will take parameters to define output filenames, or have systematic ways of writing FASTA files and defining inputs. You will have to look at the command line interface of each of those tools to figure it out (for example, sratools [here](https://github.com/ncbi/sra-tools/wiki/08.-prefetch-and-fasterq-dump), bbduk.sh [here](https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/bbduk-guide/) and spades [here](http://cab.spbu.ru/files/release3.12.0/manual.html)). an example would be as follows:
  
  ```
  #!/usr/bin/bash
  SRA=$1
  
  # get reads
  prefetch $SRA  # writes to a known path locally
  fasterq-dump $SRA --split-3  # looks for path_to_sra_file and outputs it. assumes you are splitting.
  
  # trim reads
  bbduk.sh in1=$SRA''_1.fastq in2=$SRA''_2.fastq out1=$SRA''_clean_1.fq out2=$SRA''_clean_2.fq
  
  # re-QC the files- this step does not feed into the other steps!!
  fastqc $SRA''*.fastq $SRA''*.fq  # runs fastqc on all fastq files
  
  # run assembly
  spades.py -o my_assembly --pe1 $SRA''_clean_1.fq --pe2 $SRA''_clean_2.fq
  
  ```
  
  and so on.
  
  I suppose that from a formal perspective it's better to use a workflow engine as username_2 username_2 says, but with such a straightforward workflow I'd say start simple and get more elaborate only if you need to; you'd spend 10x as long just figuring out the workflow language. This is ok if it's a goal to learn it or if you already know it, but if you just want some results putting it into a script is fine.
  
  > **Comments:**
  > **username_4**: The other virtue of putting everything in a script is every command line option is documented.  
  > **username_5**: All workflow management systems I have ever worked with also document all parameters (and much more) meticulously.  
  
  ---
  
  