Answers.Body: <p>Your samples end with <code>_1.fastq</code> and <code>_2.fastq</code>, so you need to change this:</p>
  
  <pre><code>-1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq
  </code></pre>
  
  <p>to this:</p>
  
  <pre><code>-1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-23T11:51:14.670Z
Answers.Text: Thanq. ya this was the only error in my code.
Answers.UserDisplayName: stack_learner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T06:27:14.307Z
Answers.Text: Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?
Answers.UserDisplayName: stack_learner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T06:36:21.350Z
Answers.Text: Untar the files and use a workflow manager like snakemake. BTW, make sure you have sufficient resources to process many samples in parallel.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T06:43:09.847Z
Answers.Text: But as soon as Untar done for one file it should be the input for hisat2. Yes, I'm using 20 cpu cores, 1 week time and tmp 500G, memory usage 4G.
Answers.UserDisplayName: stack_learner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T06:44:05.930Z
Answers.Text: Then put the untar command into something like Snakemake too.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T07:40:15.860Z
Answers.Text: Never used a snakemake till now. May I know how I can give that?
Answers.UserDisplayName: stack_learner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T07:40:51.380Z
Answers.Text: Please google such things and post a new question if you have further questions.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-23T09:31:54.723Z
Answers.LastActivityDate: 2018-04-23T09:31:54.723Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>This should work for you:</p>
  
  <pre><code>for i in *.tar.gz
  do
      dir2="/destination/directory"
      tar xvzf $i -C $TMPDIR
      files=($TMPDIR/*)
      module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - &gt; ${dir2}/$i.bam
  done
  </code></pre>
  
  <p>(If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).</p>
  
  <p>Edit:</p>
  
  <p>If you want to run on all files at the same time, the easiest way is <code>find</code> / <code>xargs</code></p>
  
  <p>Change you script to this:</p>
  
  <pre><code>for i in $@
  do
      dir2="/destination/directory"
      tar xvzf $i -C $TMPDIR
      files=($TMPDIR/*)
      module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - &gt; ${dir2}/$i.bam
  done
  </code></pre>
  
  <p>Then run it like this:</p>
  
  <pre><code>find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT
  </code></pre>
  
  <p>This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use <code>gnu parallel</code> or something. As <code>@Ian Sudbury</code> says, pipeline software can take care of all these details for you.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-23T09:49:18.773Z
Answers.Text: Definately +1 for putting all this in some sort of pipeline. Many of these  will now handle automated distribution of many files to your SLURM cluster. Job tracking. Restarts after errors etc.
Answers.UserDisplayName: Ian Sudbery
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T06:26:52.177Z
Answers.Text: @heathobrien Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?
Answers.UserDisplayName: stack_learner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T07:12:27.417Z
Answers.Text: see updated answer
Answers.UserDisplayName: heathobrien
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-30T11:48:32.263Z
Answers.Text: Even I haven't worked on SLURM before. Do you think if I give #SBATCH --array=1-6%3 will work? lets say I have 6 tar.gz files so 3 jobs at a time.
Answers.UserDisplayName: stack_learner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-23T09:33:31.583Z
Answers.LastActivityDate: 2018-04-30T07:12:16.343Z
Answers.LastEditDate: 2018-04-30T07:12:16.343Z
Answers.UserDisplayName: heathobrien
Body: <p>Original tar.gz files look like below:</p>
  
  <pre><code>UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
  UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
  UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
  UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz
  </code></pre>
  
  <p>Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:</p>
  
  <pre><code>xargs -n2 mv -i &lt;&lt;&lt;"$(sed '1d; s/$/.tar.gz/' names.txt)"
  
  Tar.gz_filenames       Sample_Names
  UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
  UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
  UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
  UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4
  </code></pre>
  
  <p>After changing the names, looks like below:</p>
  
  <pre><code>Sample1.tar.gz
  Sample2.tar.gz
  Sample3.tar.gz
  Sample4.tar.gz
  </code></pre>
  
  <p>Using the tar.gz files after changing names I used the below code:</p>
  
  <pre><code>for i in /doc/*.tar.gz
  do
      tar xvzf $i -C $TMPDIR
      for sample in $TMPDIR/*1.fastq
      do
          dir2="/destination/directory"
          base=$(basename $sample ".1.fastq")
          module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq | samtools view -Sb - &gt; ${dir2}/${base}.bam
      done
  done
  </code></pre>
  
  <p>When I run the above code, the output is like below:</p>
  
  <pre><code>150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq
  150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq
  150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq
  150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq
  140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq
  140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq
  110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq
  110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq
  Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping...
  Error: No input read files were valid
  (ERR): hisat2-align exited with value 1
  Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping...
  Error: No input read files were valid
  (ERR): hisat2-align exited with value 1
  Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping...
  Error: No input read files were valid
  (ERR): hisat2-align exited with value 1
  </code></pre>
  
  <p>I dont mins whatever the name of fastq's in <code>$TMPDIR</code> but from <code>$TMPDIR</code> hisat2 should use those fastq's and give output bam files in /destination like below:</p>
  
  <pre><code>Sample1.bam
  Sample2.bam
  Sample3.bam
  Sample4.bam
  </code></pre>
  
  <p>I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop)</p>
  
Tags: <rna-seq><file-formats><fastq><linux>
Title: Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?
ThreadText: # Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?
  
  (Asked by: username_0 on 2018-04-23)
  
  Original tar.gz files look like below:
  
  ```
  UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
  UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
  UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
  UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz
  
  ```
  
  Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:
  
  ```
  xargs -n2 mv -i <<<"$(sed '1d; s/$/.tar.gz/' names.txt)"
  
  Tar.gz_filenames       Sample_Names
  UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
  UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
  UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
  UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4
  
  ```
  
  After changing the names, looks like below:
  
  ```
  Sample1.tar.gz
  Sample2.tar.gz
  Sample3.tar.gz
  Sample4.tar.gz
  
  ```
  
  Using the tar.gz files after changing names I used the below code:
  
  ```
  for i in /doc/*.tar.gz
  do
      tar xvzf $i -C $TMPDIR
      for sample in $TMPDIR/*1.fastq
      do
          dir2="/destination/directory"
          base=$(basename $sample ".1.fastq")
          module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq | samtools view -Sb - > ${dir2}/${base}.bam
      done
  done
  
  ```
  
  When I run the above code, the output is like below:
  
  ```
  150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq
  150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq
  150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq
  150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq
  140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq
  140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq
  110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq
  110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq
  Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping...
  Error: No input read files were valid
  (ERR): hisat2-align exited with value 1
  Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping...
  Error: No input read files were valid
  (ERR): hisat2-align exited with value 1
  Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping...
  Error: No input read files were valid
  (ERR): hisat2-align exited with value 1
  
  ```
  
  I dont mins whatever the name of fastq's in `$TMPDIR` but from `$TMPDIR` hisat2 should use those fastq's and give output bam files in /destination like below:
  
  ```
  Sample1.bam
  Sample2.bam
  Sample3.bam
  Sample4.bam
  
  ```
  
  I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop)
  
  **Tags:** rna-seq, file-formats, fastq, linux
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-04-23. Score: 7 | Accepted answer
  
  This should work for you:
  
  ```
  for i in *.tar.gz
  do
      dir2="/destination/directory"
      tar xvzf $i -C $TMPDIR
      files=($TMPDIR/*)
      module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
  done
  
  ```
  
  (If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).
  
  Edit:
  
  If you want to run on all files at the same time, the easiest way is `find` / `xargs`
  
  Change you script to this:
  
  ```
  for i in $@
  do
      dir2="/destination/directory"
      tar xvzf $i -C $TMPDIR
      files=($TMPDIR/*)
      module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
  done
  
  ```
  
  Then run it like this:
  
  ```
  find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT
  
  ```
  
  This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use `gnu parallel` or something. As `@Ian Sudbury` says, pipeline software can take care of all these details for you.
  
  > **Comments:**
  > **username_3**: Definately +1 for putting all this in some sort of pipeline. Many of these  will now handle automated distribution of many files to your SLURM cluster. Job tracking. Restarts after errors etc.  
  > **username_0**: @username_2 Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?  
  > **username_2**: see updated answer  
  > **username_0**: Even I haven't worked on SLURM before. Do you think if I give #SBATCH --array=1-6%3 will work? lets say I have 6 tar.gz files so 3 jobs at a time.  
  
  ---
  
  ### Answer by username_1 on 2018-04-23. Score: 1
  
  Your samples end with `_1.fastq` and `_2.fastq`, so you need to change this:
  
  ```
  -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq
  
  ```
  
  to this:
  
  ```
  -1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq
  
  ```
  
  > **Comments:**
  > **username_0**: Thanq. ya this was the only error in my code.  
  > **username_0**: Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?  
  > **username_1**: Untar the files and use a workflow manager like snakemake. BTW, make sure you have sufficient resources to process many samples in parallel.  
  > **username_0**: But as soon as Untar done for one file it should be the input for hisat2. Yes, I'm using 20 cpu cores, 1 week time and tmp 500G, memory usage 4G.  
  > **username_1**: Then put the untar command into something like Snakemake too.  
  > **username_0**: Never used a snakemake till now. May I know how I can give that?  
  > **username_1**: Please google such things and post a new question if you have further questions.  
  
  ---
  
  