Answers.Body: <p>All you need is <code>cat</code>. You won't find any better tool for a simple job like this. Just run:</p>
  
  <pre><code>cat NA24694_GCCAAT_L001_R1*fastq.gz &gt; EA00694_GCCAAT_L001_R1.fastq.gz
  cat NA24694_GCCAAT_L001_R2*fastq.gz &gt; EA00694_GCCAAT_L001_R2.fastq.gz
  cat NA24694_GCCAAT_L001_R3*fastq.gz &gt; EA00694_GCCAAT_L002_R1.fastq.gz
  cat NA24694_GCCAAT_L001_R4*fastq.gz &gt; EA00694_GCCAAT_L002_R2.fastq.gz
  </code></pre>
  
  <p>That should work just fine. You say you tried <code>cat</code> and it "didn't work" but since you don't tell us how it failed, I can't really help. The only issues I can think of is that either there are too many files (as in several hundred thousand, whatever the value returned by <code>getconf ARG_MAX</code> on your system which is 2097152 on mine) or, more likely, you are running out of disk space. </p>
  
  <p>If it's a disk space issue, you might be able to get around it by adding each file and then deleting it:</p>
  
  <pre><code>for i in 1 2 3 4; do 
      for file in NA24694_GCCAAT_L001_R<span class="math-container">${i}_*fastq.gz; do
          cat "$</span>file" &gt;&gt; EA00694_GCCAAT_L001_R<span class="math-container">${i}.fastq.gz &amp;&amp; rm "$</span>file"
      done
  done
  </code></pre>
  
  <p><strong><em>IMPORTANT</em></strong>: the command above is destructive. It will delete each file after it has been added to the new one. If everything works fine that's not a problem, but if I've made a mistake or your file names are slightly different you might lose your data. So I strongly urge you to simply run the <code>cat</code> commands in the first section on a machine with enough disk space to deal with them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-17T04:32:30.920Z
Answers.Text: I was using cat command manually by adding to file names to it  I had these types of files    EA13698_AGTAAT_L001_R1_001.fastq.gz  
  EA13698_AGTAAT_L001_R2_032.fastq.gz  
  EA13698_AGTAAT_L002_R1_003.fastq.gz  
  EA13698_AGTAAT_L002_R2_037.fastq.gz  
  
  I had these the four types of files, in each group it had more than 50 files so I couldnt keep on adding names to cat command.   eg "cat EA13698_AGTAAT_L001_R1_001.fastq.gz 
  EA13698_AGTAAT_L001_R1_002.fastq.gz 
   .... > EA13698_AGTAAT_L001_R1.fastq.gz. manually doing this it takes time so I was searching for script which will do looping.
Answers.UserDisplayName: Nitha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-17T06:42:57.080Z
Answers.Text: No need to loop, just do them all at once with globbing. 50 files won't be too long and you won't have to type them out.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-17T10:22:23.603Z
Answers.Text: IMO the way I would do it is with ">>" (append) rather than brute forcing the concatenation with ">" (over-ride) ... which is what the loop is about. I like the shell loop.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-17T10:38:10.420Z
Answers.Text: As said by @terdon, I had used * to group the types of file in 'cat'.. it worked. Thanks all.
Answers.UserDisplayName: Nitha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T16:49:36.707Z
Answers.LastActivityDate: 2020-02-27T10:05:59.570Z
Answers.LastEditDate: 2020-02-27T10:05:59.570Z
Answers.UserDisplayName: terdon
Answers.Body: <p>We sequence DNA on daily basis. Like <code>terdon</code> said, you can use <code>cat</code> to group all the FastQ of all your sample whether they are bgzipped or not like so:</p>
  
  <pre><code>root@slurm1: ~  # ls
  am224_S6_L001_R1_001.fastq.gz
  am224_S6_L001_R2_002.fastq.gz
  am224_S6_L002_R1_003.fastq.gz
  am224_S6_L002_R2_004.fastq.gz
  am232_S10_L001_R1_001.fastq.gz
  am232_S10_L001_R2_002.fastq.gz
  am232_S10_L002_R1_003.fastq.gz
  am232_S10_L002_R2_004.fastq.gz
  </code></pre>
  
  <p>You could do:</p>
  
  <pre><code># initiate am empty array
  listSamples=()
  
  # loop to add the unique samples to an array
  for i in *.fastq.gz;
  do
      if [[ "<span class="math-container">$sampleName" =~ "$</span>{listSamples[@]}" ]]; then
          sampleName=<span class="math-container">$(echo $</span>i | cut -d'_' -f1)
          listSamples+=("$sampleName")
      fi
  done
  
  # loop in the newly created array of samples and group them by reads number
  for i in "<span class="math-container">${listSamples[@]}"
  do
      cat "$</span>i"_*_R1_* &gt; "<span class="math-container">$i"_1.fastq.gz
      cat "$</span>i"_*_R2_* &gt; "$i"_2.fastq.gz
  done
  </code></pre>
  
  <p>However, I highly suggest you to compress your files as <code>.bz2</code> to save storage as much as possible. You can achieve this using the multithreaded tool <a href="http://lbzip2.org/" rel="nofollow noreferrer">lbzip2</a>. Assuming that you have 10 CPUs available, you can do:</p>
  
  <pre><code>for i in "${listSamples[@]}"
  do
      bzcat "$i"_*_R1_* | lbzip2 -n 10 --best &gt; "$i"_1.fastq.bz2
      bzcat "$i"_*_R2_* | lbzip2 -n 10 --best &gt; "$i"_2.fastq.bz2
  done
  </code></pre>
  
  <p>Hope this helps !</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-20T16:57:40.717Z
Answers.Text: I like the bz2 storage suggestion
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-20T16:32:47.027Z
Answers.LastActivityDate: 2019-05-20T16:32:47.027Z
Answers.UserDisplayName: user324810
Body: <p>I have a large number of .fastq.gz files of different lane and reads. I have to merge them each reads group files into single .fastq.gz files.</p>
  
  <p>**eg:</p>
  
  <p><strong>1st type</strong><br>
  NA24694_GCCAAT_L001_R1_001.fastq.gz<br>
  NA24694_GCCAAT_L001_R1_002.fastq.gz<br>
  …....<br>
  …....<br>
  NA24694_GCCAAT_L001_R1_033.fastq.gz</p>
  
  <p><strong>2nd type</strong><br>
  NA24694_GCCAAT_L001_R2_001.fastq.gz<br>
  NA24694_GCCAAT_L001_R2_002.fastq.gz<br>
  …....<br>
  …....<br>
  NA24694_GCCAAT_L001_R2_027.fastq.gz</p>
  
  <p><strong>3rd type</strong><br>
  NA24694_GCCAAT_L002_R1_001.fastq.g<br>
  NA24694_GCCAAT_L002_R1_002.fastq.g<br>
  …....<br>
  …....<br>
  NA24694_GCCAAT_L002_R1_040.fastq.g    </p>
  
  <p><strong>4th type</strong><br>
  NA24694_GCCAAT_L002_R2_001.fastq.gz<br>
  NA24694_GCCAAT_L002_R2_002.fastq.gz<br>
  …....<br>
  …....<br>
  NA24694_GCCAAT_L002_R2_040.fastq.gz</p>
  
  <p>so now i need to merge all the corresponding files into single file.  </p>
  
  <p><strong>Output:</strong><br>
  EA00694_GCCAAT_L001_R1.fastq.gz<br>
  EA00694_GCCAAT_L001_R2.fastq.gz<br>
  EA00694_GCCAAT_L002_R1.fastq.gz<br>
  EA00694_GCCAAT_L002_R2.fastq.gz  </p>
  
  <p>I tried ""cat'" but its not possible to do for large files. Could any one help me to run in script parallel in perl or python or shell. or any linux command which can be used for large file.</p>
  
  <p>Thanks all.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-16T10:32:33.190Z
Comments.Text: Why is `cat` not possible for large files, please tell more about the size and number of files and/or the error you get when running the `cat` command.
Comments.UserDisplayName: benn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-16T10:44:52.650Z
Comments.Text: the folder size 40gb.. there are >200 files for each type.. so I couldn't do it manually thats why I am looking for scripts or any efficient command
Comments.UserDisplayName: Nitha
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-16T10:47:05.317Z
Comments.Text: Did you try `cat NA24694_GCCAAT_L001_R1_???.fastq.gz > EA00694_GCCAAT_L001_R1.fastq.gz`?
Comments.UserDisplayName: benn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-16T12:29:05.037Z
Comments.Text: You need to dump your data on an external harddrive with sufficient. If you run out of harddrive it will throw an exception, or destroy the data. You are requesting a solution in higher order programming, I don't think its needed, just carefully crafted key words to build the file up in stages rather than all at once.
Comments.UserDisplayName: M__
Tags: <bash><shell><data-management>
Title: How to merge .fastq.qz files into a single .fastq.gz with their same id without losing any content in parallel
ThreadText: # How to merge .fastq.qz files into a single .fastq.gz with their same id without losing any content in parallel
  
  (Asked by: username_0 on 2019-05-16)
  
  I have a large number of .fastq.gz files of different lane and reads. I have to merge them each reads group files into single .fastq.gz files.
  
  \*\*eg:
  
  **1st type**  
  NA24694\_GCCAAT\_L001\_R1\_001.fastq.gz  
  NA24694\_GCCAAT\_L001\_R1\_002.fastq.gz  
  …....  
  …....  
  NA24694\_GCCAAT\_L001\_R1\_033.fastq.gz
  
  **2nd type**  
  NA24694\_GCCAAT\_L001\_R2\_001.fastq.gz  
  NA24694\_GCCAAT\_L001\_R2\_002.fastq.gz  
  …....  
  …....  
  NA24694\_GCCAAT\_L001\_R2\_027.fastq.gz
  
  **3rd type**  
  NA24694\_GCCAAT\_L002\_R1\_001.fastq.g  
  NA24694\_GCCAAT\_L002\_R1\_002.fastq.g  
  …....  
  …....  
  NA24694\_GCCAAT\_L002\_R1\_040.fastq.g
  
  **4th type**  
  NA24694\_GCCAAT\_L002\_R2\_001.fastq.gz  
  NA24694\_GCCAAT\_L002\_R2\_002.fastq.gz  
  …....  
  …....  
  NA24694\_GCCAAT\_L002\_R2\_040.fastq.gz
  
  so now i need to merge all the corresponding files into single file.
  
  **Output:**  
  EA00694\_GCCAAT\_L001\_R1.fastq.gz  
  EA00694\_GCCAAT\_L001\_R2.fastq.gz  
  EA00694\_GCCAAT\_L002\_R1.fastq.gz  
  EA00694\_GCCAAT\_L002\_R2.fastq.gz
  
  I tried ""cat'" but its not possible to do for large files. Could any one help me to run in script parallel in perl or python or shell. or any linux command which can be used for large file.
  
  Thanks all.
  
  **Tags:** bash, shell, data-management
  
  > **Comments:**
  > **username_1**: Why is `cat` not possible for large files, please tell more about the size and number of files and/or the error you get when running the `cat` command.  
  > **username_0**: the folder size 40gb.. there are >200 files for each type.. so I couldn't do it manually thats why I am looking for scripts or any efficient command  
  > **username_1**: Did you try `cat NA24694_GCCAAT_L001_R1_???.fastq.gz > EA00694_GCCAAT_L001_R1.fastq.gz`?  
  > **username_2**: You need to dump your data on an external harddrive with sufficient. If you run out of harddrive it will throw an exception, or destroy the data. You are requesting a solution in higher order programming, I don't think its needed, just carefully crafted key words to build the file up in stages rather than all at once.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-05-16. Score: 10 | Accepted answer
  
  All you need is `cat`. You won't find any better tool for a simple job like this. Just run:
  
  ```
  cat NA24694_GCCAAT_L001_R1*fastq.gz > EA00694_GCCAAT_L001_R1.fastq.gz
  cat NA24694_GCCAAT_L001_R2*fastq.gz > EA00694_GCCAAT_L001_R2.fastq.gz
  cat NA24694_GCCAAT_L001_R3*fastq.gz > EA00694_GCCAAT_L002_R1.fastq.gz
  cat NA24694_GCCAAT_L001_R4*fastq.gz > EA00694_GCCAAT_L002_R2.fastq.gz
  
  ```
  
  That should work just fine. You say you tried `cat` and it "didn't work" but since you don't tell us how it failed, I can't really help. The only issues I can think of is that either there are too many files (as in several hundred thousand, whatever the value returned by `getconf ARG_MAX` on your system which is 2097152 on mine) or, more likely, you are running out of disk space.
  
  If it's a disk space issue, you might be able to get around it by adding each file and then deleting it:
  
  ```
  for i in 1 2 3 4; do 
      for file in NA24694_GCCAAT_L001_R${i}_*fastq.gz; do
          cat "$file" >> EA00694_GCCAAT_L001_R${i}.fastq.gz && rm "$file"
      done
  done
  
  ```
  
  ***IMPORTANT***: the command above is destructive. It will delete each file after it has been added to the new one. If everything works fine that's not a problem, but if I've made a mistake or your file names are slightly different you might lose your data. So I strongly urge you to simply run the `cat` commands in the first section on a machine with enough disk space to deal with them.
  
  > **Comments:**
  > **username_0**: I was using cat command manually by adding to file names to it  I had these types of files    EA13698_AGTAAT_L001_R1_001.fastq.gz   EA13698_AGTAAT_L001_R2_032.fastq.gz   EA13698_AGTAAT_L002_R1_003.fastq.gz   EA13698_AGTAAT_L002_R2_037.fastq.gz    I had these the four types of files, in each group it had more than 50 files so I couldnt keep on adding names to cat command.   eg "cat EA13698_AGTAAT_L001_R1_001.fastq.gz  EA13698_AGTAAT_L001_R1_002.fastq.gz   .... > EA13698_AGTAAT_L001_R1.fastq.gz. manually doing this it takes time so I was searching for script which will do looping.  
  > **username_4**: No need to loop, just do them all at once with globbing. 50 files won't be too long and you won't have to type them out.  
  > **username_2**: IMO the way I would do it is with ">>" (append) rather than brute forcing the concatenation with ">" (over-ride) ... which is what the loop is about. I like the shell loop.  
  > **username_0**: As said by @username_3, I had used * to group the types of file in 'cat'.. it worked. Thanks all.  
  
  ---
  
  ### Answer by username_5 on 2019-05-20. Score: 3
  
  We sequence DNA on daily basis. Like `username_3` said, you can use `cat` to group all the FastQ of all your sample whether they are bgzipped or not like so:
  
  ```
  root@slurm1: ~  # ls
  am224_S6_L001_R1_001.fastq.gz
  am224_S6_L001_R2_002.fastq.gz
  am224_S6_L002_R1_003.fastq.gz
  am224_S6_L002_R2_004.fastq.gz
  am232_S10_L001_R1_001.fastq.gz
  am232_S10_L001_R2_002.fastq.gz
  am232_S10_L002_R1_003.fastq.gz
  am232_S10_L002_R2_004.fastq.gz
  
  ```
  
  You could do:
  
  ```
  # initiate am empty array
  listSamples=()
  
  # loop to add the unique samples to an array
  for i in *.fastq.gz;
  do
      if [[ "$sampleName" =~ "${listSamples[@]}" ]]; then
          sampleName=$(echo $i | cut -d'_' -f1)
          listSamples+=("$sampleName")
      fi
  done
  
  # loop in the newly created array of samples and group them by reads number
  for i in "${listSamples[@]}"
  do
      cat "$i"_*_R1_* > "$i"_1.fastq.gz
      cat "$i"_*_R2_* > "$i"_2.fastq.gz
  done
  
  ```
  
  However, I highly suggest you to compress your files as `.bz2` to save storage as much as possible. You can achieve this using the multithreaded tool [lbzip2](http://lbzip2.org/). Assuming that you have 10 CPUs available, you can do:
  
  ```
  for i in "${listSamples[@]}"
  do
      bzcat "$i"_*_R1_* | lbzip2 -n 10 --best > "$i"_1.fastq.bz2
      bzcat "$i"_*_R2_* | lbzip2 -n 10 --best > "$i"_2.fastq.bz2
  done
  
  ```
  
  Hope this helps !
  
  > **Comments:**
  > **username_2**: I like the bz2 storage suggestion  
  
  ---
  
  