Answers.Body: <p>Your only option here is to first split the fastq files into small chunks, process each and then merge them back together. Alternatively, find something else that runs faster.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T14:03:08.770Z
Answers.LastActivityDate: 2017-05-19T14:03:08.770Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>Here is a bash script and GNU <code>parallel</code> along with some other tools to split-apply-combine the data. It may or may not work out of the box, but it's not far off and it should give the general idea of how one way of how to approach this.</p>
  
  
  
  <pre><code>#!/bin/bash
  
  # ====== USER PROVIDED PARAMETERS ========
  N=20 # number of cores to use
  ngsfilter="rawdata_scandinavia.ngsfilter"
  unidentified="unidentified_samples.fastq"
  
  R1="rawdata_scandinavia_R1.fastq"
  R2="rawdata_scandinavia_R2.fastq"
  # ==== END USER PROVIDED PARAMETERS =====
  
  # fastqutils available @ http://ngsutils.org/modules/fastqutils
  fastqutils split $R1 to_illumina_R1_ $N &amp;
  fastqutils split $R2 to_illumina_R2_ $N &amp;
  wait
  
  R1=$( ls | grep -P "to_illumina_R1_" )
  R2=$( ls | grep -P "to_illumina_R2_" )
  
  # Do alignment of two reads.
  for i in $( seq $N )
  do
      partR1=$( ls | grep -P "^to_illumina_R1_\\.$i\\.fastq$" )
      partR2=$( ls | grep -P "^to_illumina_R2_\\.$i\\.fastq$" )
      illuminapairedend -r $partR1 $partR2 | tee to_ngsfilter_$i.fastq | obiannotate -S goodAli:'"Align" if score&gt;40.00 else "Bad"' | obisplit -t goodAli -p to_ngsfilter_$i. &amp;
  done
  
  # Remove intermediate aligned files.
  echo $R1 | xargs rm -r
  echo $R2 | xargs rm -r
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  input=$( ls | grep -P "to_ngsfilter_[0-9]+\\.Align\\.fastq" )
  
  # For some reasons, files need to be "touched" in order to work. Perhaps
  # it's not closing them...
  touch $unidentified
  
  touchedbyangel=$( ls | grep -P "to_ngsfilter_"  )
  for i in $touchedbyangel
  do
      touch $i
  done
  
  parallel -j$N --result filtered_{} ngsfilter -t $ngsfilter -u $unidentified {} ::: $input
  
  # Remove intermediate results.
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  ls | grep -P "filtered_.*\\.fastq$" | xargs cat &gt; filtered_data.fastq
  
  # remove intermediate results to conserve space
  rm to_ngsfilter*
  
  # Split reads by locus.
  filtered=$( ls | grep -P "^filtered_"  )
  parallel -j$N --result to_split_{} obisplit -p MS.PCR_ -t experiment {} ::: $filtered
  
  ls | grep -P "\\.err$|\\.seq$" | xargs rm -r
  ls | grep -P "^to_split_" | xargs rm -r
  
  # Wasn't able to parallelize this, so it's just pushing tasks into the background.
  obiuniq -m sample MS.PCR_UA_MxRout1_03.fastq &gt; MS.PCR_UA_MxRout1_03.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_06.fastq &gt; MS.PCR_UA_MxRout1_06.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_14.fastq &gt; MS.PCR_UA_MxRout1_14.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_16.fastq &gt; MS.PCR_UA_MxRout1_16.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_17.fastq &gt; MS.PCR_UA_MxRout1_17.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_25.fastq &gt; MS.PCR_UA_MxRout1_25.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_51.fastq &gt; MS.PCR_UA_MxRout1_51.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_57.fastq &gt; MS.PCR_UA_MxRout1_57.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_63.fastq &gt; MS.PCR_UA_MxRout1_63.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_64.fastq &gt; MS.PCR_UA_MxRout1_64.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_65.fastq &gt; MS.PCR_UA_MxRout1_65.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_67.fastq &gt; MS.PCR_UA_MxRout1_67.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_68.fastq &gt; MS.PCR_UA_MxRout1_68.uniq.fasta &amp;
  obiuniq -m sample MS.PCR_UA_MxRout1_ZF.fastq &gt; MS.PCR_UA_MxRout1_ZF.uniq.fasta &amp;
  wait
  
  uniqfiles=$( ls | grep -P ".{2}+\\.uniq\\.fasta$" )
  
  for i in $uniqfiles
  do
      # this section removes the file suffix
      # https://stackoverflow.com/questions/125281/how-do-i-remove-the-file-suffix-and-path-portion-from-a-path-string-in-bash
      tab=${i%.fasta}
      tab=${tab##*/}
      obigrep -p 'count&gt;1' $i | obiannotate -k merged_sample -k count | obiannotate --length | obisort -r -k seq_length | obitab --no-definition --output-seq &gt; $tab.tab
  done
  
  ls | grep -P "\\.uniq\\.fasta" | xargs rm -r
  
  echo "Done. I think."
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-02T09:35:00.113Z
Answers.LastActivityDate: 2017-06-02T13:18:13.977Z
Answers.LastEditDate: 2017-06-02T13:18:13.977Z
Answers.UserDisplayName: Roman Luštrik
Body: <p>I have an <a href="http://datadryad.org/bitstream/handle/10255/dryad.122664/obitools_script.txt?sequence=1" rel="nofollow noreferrer">obitools script</a> (de Barba et al. 2016) that I would like to run faster. How would you run it in parallel to cut down on time?</p>
  
  <pre><code>illuminapairedend -r rawdata_scandinavia_R2.fastq rawdata_scandinavia_R1.fastq | tee rawdata_scandinavia.fastq | obiannotate -S goodAli:'"Alignement" if score&gt;40.00 else "Bad"' | obisplit -t goodAli -p rawdata_scandinavia.
  touch rawdata_scandinavia.Bad.fastq
  touch rawdata_scandinavia.Alignement.fastq
  touch rawdata_scandinavia.fastq
  ngsfilter -t rawdata_scandinavia.ngsfilter -u rawdata_scandinavia.unidentified.fastq rawdata_scandinavia.Alignement.fastq &gt; rawdata_scandinavia.filtered.fastq
  obisplit -p MICROSAT.PCR_ -t experiment rawdata_scandinavia.filtered.fastq
  </code></pre>
  
  <p>Can someone describe what tools and steps one needs to take to run this job in parallel, given that the functions are not designed to be run in parallel out of the box?</p>
  
  <p>De Barba, M., Miquel, C., Lobréaux, S., Quenette, P. Y., Swenson, J. E., &amp; Taberlet, P. (2016). High-throughput microsatellite genotyping in ecology: improved accuracy, efficiency, standardization and success with low-quantity and degraded DNA. Molecular Ecology Resources, 1–16. <a href="https://doi.org/10.1111/1755-0998.12594" rel="nofollow noreferrer">https://doi.org/10.1111/1755-0998.12594</a></p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-01T22:12:24.330Z
Comments.Text: Bioinformatics encompasses biology, statistics, and computer science. Parallel processing is a common need in bioinformatics, and can be a challenging task to implement depending on the particular programs that are used. The programs used here in the example script are specific to bioinformatics, and answers in stack overflow might be lower quality due to the inexperience of the community in working with bioinformatics programs.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T08:15:25.103Z
Comments.Text: @gringer this question was downvoted and closed because it's a programming question. Fast forward a few weeks later, [they're suddenly fine](https://bioinformatics.stackexchange.com/questions/361/what-is-the-fastest-way-to-calculate-the-number-of-unknown-nucleotides-in-fasta).
Comments.UserDisplayName: Roman Luštrik
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T08:25:47.657Z
Comments.Text: Pure programming questions (as was the impression from reading only the question title for this) are better placed in SO. However, this question is about a specific type of parallel processing that is associated with a particular bioinformatic tool. I can imagine that this question might be closed in SO for offtopic reasons.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T08:30:56.183Z
Comments.Text: @gringer I hope we can get one more vote. I've solved the problem in the mean time and would like to offer my fruits of labor to anybody who might come up with this kind of a problem in the future.
Comments.UserDisplayName: Roman Luštrik
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T08:50:51.697Z
Comments.Text: @RomanLuštrik the FASTA/FASTQ question has now been updated with more context.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-19T14:02:01.897Z
Comments.Text: I think this is a more general programming question and so would be more suited for stack overflow
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-05-19T14:44:49.407Z
Comments.Text: Divide & conquer!
Comments.UserDisplayName: Kamil S Jaron
Tags: <ngs><linux>
Title: Parallel processing of scripts that use obitools
ThreadText: # Parallel processing of scripts that use obitools
  
  (Asked by: username_0 on 2017-05-19)
  
  I have an [obitools script](http://datadryad.org/bitstream/handle/10255/dryad.122664/obitools_script.txt?sequence=1) (de Barba et al. 2016) that I would like to run faster. How would you run it in parallel to cut down on time?
  
  ```
  illuminapairedend -r rawdata_scandinavia_R2.fastq rawdata_scandinavia_R1.fastq | tee rawdata_scandinavia.fastq | obiannotate -S goodAli:'"Alignement" if score>40.00 else "Bad"' | obisplit -t goodAli -p rawdata_scandinavia.
  touch rawdata_scandinavia.Bad.fastq
  touch rawdata_scandinavia.Alignement.fastq
  touch rawdata_scandinavia.fastq
  ngsfilter -t rawdata_scandinavia.ngsfilter -u rawdata_scandinavia.unidentified.fastq rawdata_scandinavia.Alignement.fastq > rawdata_scandinavia.filtered.fastq
  obisplit -p MICROSAT.PCR_ -t experiment rawdata_scandinavia.filtered.fastq
  
  ```
  
  Can someone describe what tools and steps one needs to take to run this job in parallel, given that the functions are not designed to be run in parallel out of the box?
  
  De Barba, M., Miquel, C., Lobréaux, S., Quenette, P. Y., Swenson, J. E., & Taberlet, P. (2016). High-throughput microsatellite genotyping in ecology: improved accuracy, efficiency, standardization and success with low-quantity and degraded DNA. Molecular Ecology Resources, 1–16. <https://doi.org/10.1111/1755-0998.12594>
  
  **Tags:** ngs, linux
  
  > **Comments:**
  > **username_2**: I think this is a more general programming question and so would be more suited for stack overflow  
  > **username_3**: Divide & conquer!  
  > **username_1**: Bioinformatics encompasses biology, statistics, and computer science. Parallel processing is a common need in bioinformatics, and can be a challenging task to implement depending on the particular programs that are used. The programs used here in the example script are specific to bioinformatics, and answers in stack overflow might be lower quality due to the inexperience of the community in working with bioinformatics programs.  
  > **username_0**: @username_1 this question was downvoted and closed because it's a programming question. Fast forward a few weeks later, [they're suddenly fine](https://bioinformatics.stackexchange.com/questions/361/what-is-the-fastest-way-to-calculate-the-number-of-unknown-nucleotides-in-fasta).  
  > **username_1**: Pure programming questions (as was the impression from reading only the question title for this) are better placed in SO. However, this question is about a specific type of parallel processing that is associated with a particular bioinformatic tool. I can imagine that this question might be closed in SO for offtopic reasons.  
  > **username_0**: @username_1 I hope we can get one more vote. I've solved the problem in the mean time and would like to offer my fruits of labor to anybody who might come up with this kind of a problem in the future.  
  > **username_1**: @username_0 the FASTA/FASTQ question has now been updated with more context.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2017-05-19. Score: 3
  
  Your only option here is to first split the fastq files into small chunks, process each and then merge them back together. Alternatively, find something else that runs faster.
  
  
  ---
  
  ### Answer by username_0 on 2017-06-02. Score: 3
  
  Here is a bash script and GNU `parallel` along with some other tools to split-apply-combine the data. It may or may not work out of the box, but it's not far off and it should give the general idea of how one way of how to approach this.
  
  ```
  #!/bin/bash
  
  # ====== USER PROVIDED PARAMETERS ========
  N=20 # number of cores to use
  ngsfilter="rawdata_scandinavia.ngsfilter"
  unidentified="unidentified_samples.fastq"
  
  R1="rawdata_scandinavia_R1.fastq"
  R2="rawdata_scandinavia_R2.fastq"
  # ==== END USER PROVIDED PARAMETERS =====
  
  # fastqutils available @ http://ngsutils.org/modules/fastqutils
  fastqutils split $R1 to_illumina_R1_ $N &
  fastqutils split $R2 to_illumina_R2_ $N &
  wait
  
  R1=$( ls | grep -P "to_illumina_R1_" )
  R2=$( ls | grep -P "to_illumina_R2_" )
  
  # Do alignment of two reads.
  for i in $( seq $N )
  do
      partR1=$( ls | grep -P "^to_illumina_R1_\\.$i\\.fastq$" )
      partR2=$( ls | grep -P "^to_illumina_R2_\\.$i\\.fastq$" )
      illuminapairedend -r $partR1 $partR2 | tee to_ngsfilter_$i.fastq | obiannotate -S goodAli:'"Align" if score>40.00 else "Bad"' | obisplit -t goodAli -p to_ngsfilter_$i. &
  done
  
  # Remove intermediate aligned files.
  echo $R1 | xargs rm -r
  echo $R2 | xargs rm -r
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  input=$( ls | grep -P "to_ngsfilter_[0-9]+\\.Align\\.fastq" )
  
  # For some reasons, files need to be "touched" in order to work. Perhaps
  # it's not closing them...
  touch $unidentified
  
  touchedbyangel=$( ls | grep -P "to_ngsfilter_"  )
  for i in $touchedbyangel
  do
      touch $i
  done
  
  parallel -j$N --result filtered_{} ngsfilter -t $ngsfilter -u $unidentified {} ::: $input
  
  # Remove intermediate results.
  ls | grep -P "\\.seq$|\\.err$" | xargs rm -r
  
  ls | grep -P "filtered_.*\\.fastq$" | xargs cat > filtered_data.fastq
  
  # remove intermediate results to conserve space
  rm to_ngsfilter*
  
  # Split reads by locus.
  filtered=$( ls | grep -P "^filtered_"  )
  parallel -j$N --result to_split_{} obisplit -p MS.PCR_ -t experiment {} ::: $filtered
  
  ls | grep -P "\\.err$|\\.seq$" | xargs rm -r
  ls | grep -P "^to_split_" | xargs rm -r
  
  # Wasn't able to parallelize this, so it's just pushing tasks into the background.
  obiuniq -m sample MS.PCR_UA_MxRout1_03.fastq > MS.PCR_UA_MxRout1_03.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_06.fastq > MS.PCR_UA_MxRout1_06.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_14.fastq > MS.PCR_UA_MxRout1_14.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_16.fastq > MS.PCR_UA_MxRout1_16.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_17.fastq > MS.PCR_UA_MxRout1_17.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_25.fastq > MS.PCR_UA_MxRout1_25.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_51.fastq > MS.PCR_UA_MxRout1_51.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_57.fastq > MS.PCR_UA_MxRout1_57.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_63.fastq > MS.PCR_UA_MxRout1_63.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_64.fastq > MS.PCR_UA_MxRout1_64.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_65.fastq > MS.PCR_UA_MxRout1_65.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_67.fastq > MS.PCR_UA_MxRout1_67.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_68.fastq > MS.PCR_UA_MxRout1_68.uniq.fasta &
  obiuniq -m sample MS.PCR_UA_MxRout1_ZF.fastq > MS.PCR_UA_MxRout1_ZF.uniq.fasta &
  wait
  
  uniqfiles=$( ls | grep -P ".{2}+\\.uniq\\.fasta$" )
  
  for i in $uniqfiles
  do
      # this section removes the file suffix
      # https://stackoverflow.com/questions/125281/how-do-i-remove-the-file-suffix-and-path-portion-from-a-path-string-in-bash
      tab=${i%.fasta}
      tab=${tab##*/}
      obigrep -p 'count>1' $i | obiannotate -k merged_sample -k count | obiannotate --length | obisort -r -k seq_length | obitab --no-definition --output-seq > $tab.tab
  done
  
  ls | grep -P "\\.uniq\\.fasta" | xargs rm -r
  
  echo "Done. I think."
  
  ```
  
  
  ---
  
  