Answers.Body: <p>A very simple approach, but one which will work with the names you show, is to remove any occurrence of <code>00N</code> (where <code>N</code> is any number) from right before the <code>.gz</code>. To do so, edit your script to:</p>
  
  <pre><code>for i in <span class="math-container">$(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
      -1 $</span>{i}R1_001.fastq.gz \
      -2 <span class="math-container">${i}R2_001.fastq.gz | 
          samtools view -bS - &gt;$</span>{i%}.bam"
  done
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-25T15:53:44.790Z
Answers.LastActivityDate: 2019-03-25T15:53:44.790Z
Answers.UserDisplayName: terdon
Body: <p>I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes <code>.gz</code> files, although I do see this warning when I run for two samples:</p>
  
  <blockquote>
    <p>Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size
    from default of 8192.</p>
  </blockquote>
  
  <p>My files are named like so.</p>
  
  <pre><code>10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
  10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
  11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
  11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz
  </code></pre>
  
  <p>The bash commands I tried are below. When I run this test command:</p>
  
  <pre><code>for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
      echo $i; 
  done
  </code></pre>
  
  <p>...I get this output.</p>
  
  <pre><code>12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 
  </code></pre>
  
  <p>When I run this:</p>
  
  <pre><code>for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
      echo $i 
      bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - &gt;${i%}.bam
  done
  </code></pre>
  
  <p>I do see one BAM for my paired end files. But as I see the second set of files which are named:</p>
  
  <pre><code>6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz
  </code></pre>
  
  <p>if I run the same script over them, I would get an error as the loop can't read the paired end file.</p>
  
  <p>So the tedious way is I can make a separate folder for these sets of data with <code>fastq-002.gz</code> and <code>fastq-003.gz</code> and can run the script by tweaking the <code>cut</code> parameter.</p>
  
  <p>But I would like to know what is the programmatic way of resolving the issue.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-25T14:24:22.263Z
Comments.Text: I strongly encourage you to use something like `snakeMake` rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud.
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-25T15:06:23.757Z
Comments.Text: Can't you just rename the files?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-25T15:13:48.730Z
Comments.Text: @terdon that's how our collaborator gave us and i dont want to mess up with the file name yes i can if i dont find any other way..
Comments.UserDisplayName: kcm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-25T15:14:31.607Z
Comments.Text: " strongly encourage you to use something like snakeMake rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud." i read a bit about it never tried it , i need to learn and give it a try
Comments.UserDisplayName: kcm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-26T18:31:33.363Z
Comments.Text: It's good practice to treat files from collaborators as read-only, so not changing the filenames is a good idea. But if the files are inconsistently named, it's a very good idea to create consistently named symlinks to those files. Well organized filenames and directory structures will make it SO NICE AND EASY to automate batch data processing.
Comments.UserDisplayName: Daniel Standage
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-27T04:37:36.497Z
Comments.Text: yes i did that finally
Comments.UserDisplayName: kcm
Tags: <read-mapping><fastq><shell>
Title: Batch alignment of inconsistently named Fastq files
ThreadText: # Batch alignment of inconsistently named Fastq files
  
  (Asked by: username_0 on 2019-03-25)
  
  I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes `.gz` files, although I do see this warning when I run for two samples:
  
  > Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size
  > from default of 8192.
  
  My files are named like so.
  
  ```
  10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
  10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
  11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
  11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz
  
  ```
  
  The bash commands I tried are below. When I run this test command:
  
  ```
  for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
      echo $i; 
  done
  
  ```
  
  ...I get this output.
  
  ```
  12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 
  
  ```
  
  When I run this:
  
  ```
  for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
      echo $i 
      bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - >${i%}.bam
  done
  
  ```
  
  I do see one BAM for my paired end files. But as I see the second set of files which are named:
  
  ```
  6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz
  
  ```
  
  if I run the same script over them, I would get an error as the loop can't read the paired end file.
  
  So the tedious way is I can make a separate folder for these sets of data with `fastq-002.gz` and `fastq-003.gz` and can run the script by tweaking the `cut` parameter.
  
  But I would like to know what is the programmatic way of resolving the issue.
  
  **Tags:** read-mapping, fastq, shell
  
  > **Comments:**
  > **username_1**: I strongly encourage you to use something like `snakeMake` rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud.  
  > **username_2**: Can't you just rename the files?  
  > **username_0**: @username_2 that's how our collaborator gave us and i dont want to mess up with the file name yes i can if i dont find any other way..  
  > **username_0**: " strongly encourage you to use something like snakeMake rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud." i read a bit about it never tried it , i need to learn and give it a try  
  > **username_3**: It's good practice to treat files from collaborators as read-only, so not changing the filenames is a good idea. But if the files are inconsistently named, it's a very good idea to create consistently named symlinks to those files. Well organized filenames and directory structures will make it SO NICE AND EASY to automate batch data processing.  
  > **username_0**: yes i did that finally  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-03-25. Score: 3 | Accepted answer
  
  A very simple approach, but one which will work with the names you show, is to remove any occurrence of `00N` (where `N` is any number) from right before the `.gz`. To do so, edit your script to:
  
  ```
  for i in $(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
      -1 ${i}R1_001.fastq.gz \
      -2 ${i}R2_001.fastq.gz | 
          samtools view -bS - >${i%}.bam"
  done
  
  ```
  
  
  ---
  
  