Answers.Body: <p>Based on the comments of the question I thought to put together this answer using <a href="https://www.gnu.org/software/coreutils/coreutils.html" rel="nofollow noreferrer">GNU coreutils</a> which should exist on pretty much all systems.</p>
  
  <p>The idea is to paste the 2 files together with <code>paste</code> (assuming they are in the same order and each line correspond to the same line in the other file).</p>
  
  <p>Since only the sequence and quality of both files should be kept, the lines starting with <code>@</code> and <code>+</code> need to be made empty for one of the files. This can be achived with sed, e.g. <code>sed 's/^@.*//'</code> empties all lines starting with <code>@</code> but keeps the empty line.</p>
  
  <p>To modify the second file on the fly, one can using <a href="http://tldp.org/LDP/abs/html/process-sub.html" rel="nofollow noreferrer">process substitution</a> <code>&lt;()</code>. 
  Everything strung together should be something like this:</p>
  
  <pre><code>file1=File_A.fastq
  file2=File_B.fastq
  paste -d "" $file1 &lt;(cat $file2 | sed 's/^@.*//' | sed 's/^+.*//') &gt; target.fastq
  </code></pre>
  
  <p>I haven't tested it thoroughly, but could be a good starting point.</p>
  
  <p>If you have a lot of files to paste, you might however want to process them first (not on the fly) using:</p>
  
  <pre><code>for file in $(ls *.fastq); do 
    cat $file | sed 's/^@.*//' | sed 's/^+.*//' &gt; ${file%.fastq}_cleaned.fastq
  done
  </code></pre>
  
  <p>This creates <code>File_A_mod.fastq</code> from <code>File_A.fastq</code> etc. 
  Best to move them in a separate directory (make sure the first one is the original instead of the _cleaned one since it serves as the base).
  You should have the following in the folder: <code>File_A.fastq File_B_cleaned.fastq File_C_cleaned.fastq ...</code>. 
  Those can be than pasted with <code>paste -d "" *.fastq &gt; target.fastq</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-21T10:21:06.363Z
Answers.LastActivityDate: 2018-08-21T11:49:46.720Z
Answers.LastEditDate: 2018-08-21T11:49:46.720Z
Answers.UserDisplayName: Sebastian Müller
Answers.Body: <p>What you actually want to do is overlap the reads, for which you can use tools like <a href="https://ccb.jhu.edu/software/FLASH/" rel="nofollow noreferrer">FLASH</a> or <a href="https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/bbmerge-guide/" rel="nofollow noreferrer">BBMerge</a> (I've had better luck with FLASH).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-15T17:04:03.567Z
Answers.Text: Thanks, the 2 sequences I want to process are NOT fw/rev or any other overlapping sequences (see new explanation above)
Answers.UserDisplayName: Martin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-15T17:10:09.210Z
Answers.Text: Ah, but at least mixcr will accept the two fastq files as they already are.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-15T17:41:08.907Z
Answers.Text: Yes, but somehow I need to keep the UMI/index from the 3rd file... If there is a way to do this with  mixcr&co, that would be perfect, but I am not aware of that possibility.
Answers.UserDisplayName: Martin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-15T17:42:37.300Z
Answers.Text: The samples should have been split according to index (I assume this is an Illumina sample index) and the UMI is typically added to the read name (I or someone else can produce a few lines of code to do that for you).
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-16T08:17:23.723Z
Answers.Text: Thanks a lot. Yes, it is Ilumina. Unfortunately sequences are not split for index, and index +UMI are not included in filenames (will add examples as soon as I get faster internet), but Migec can list each index in a barcode file and demultiplex according to index, if supplied with 2 (not 3) files, with UMI+index added to e.g. read 1... (used to work perfectly with this pipeliine as long as we had index+UMI added to read 1)...
Answers.UserDisplayName: Martin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-16T08:19:40.323Z
Answers.Text: Make your life easier and ask your sequencing provider to just give you the files in a standard format (e.g., split by index and with the UMIs in the header). Darn near every tool expects exactly that.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-15T17:02:19.957Z
Answers.LastActivityDate: 2018-08-15T17:02:19.957Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I'd like to fuse ~50 pairs of corresponding fastq files <em>on per sequence bases</em>. I'd be happy for a solution as a Linux script, R script (if feasible with huge files), or any free software that runs preferably under Debian Linux (otherwise Windows). Aim is preprocessing for a Linux pipeline for NGS data. Ideally both the sequence and quality data are fused.</p>
  
  <p>The <strong>aim</strong> of this request is the following: there is an existing pipeline for downstream mapping of immunoreceptor sequences (migec/mixcr/vdjtools etc.). Previously we got Illumina reads (read 1, read 2 = reverse), with the index and UMI sequences attached to one of the reads => can be alligned and processed to habe 1 biological useful sequence with UMI and index at one end. NOW we get read 1, read 2 and seperately "read 3" which is just the UMI+Index reads, and which was attached by the core facility to read 1 using software we do not have access to. Since the previous pipeline works well, we prefere to use it in the future, but are open for suggestions how to use the index reads with migec/mixcr/vdjtools otherwise.</p>
  
  <p>Each of the 2 files contains the same number (millions) of short (~300 bp) and corresponding DNA reads:</p>
  
  <p>File_A.fastq:</p>
  
  <ul>
  <li>@record1 / aaa... / +... / ...</li>
  <li>@record2 / ...</li>
  <li>@record3 / ...</li>
  <li>... (millions)</li>
  </ul>
  
  <p>File_B.fastq:</p>
  
  <ul>
  <li>@record1 / ttt... / +... / ...</li>
  <li>@record2 / ...</li>
  <li>@record3 / ...</li>
  <li>... (millions)</li>
  </ul>
  
  <p>The target file should look like:</p>
  
  <ul>
  <li>@record1 / aaa...ttt... / +... / ...</li>
  <li>@record2 / ...</li>
  <li>@record3 / ...</li>
  <li>... (millions)</li>
  </ul>
  
  <p>(of course the aaa...ttt... is just an example)</p>
  
  <p>I was looking for a solution on Stackexchange and Google, but the solutions, I found do not work for me because</p>
  
  <ul>
  <li>manual processing is not an option given the number of sequences</li>
  <li>I need to fuse the files on a <strong>per sequence</strong> basis, i.e. a simple Linux command like cat file1 file2 is not what I need (as far as I understand, this would give me (record1_A record2_A.... record1B record2_B).</li>
  <li>R, package shortread, readFastq + writeFastq would be perfect, but cannot handle such large numbers of input lines</li>
  </ul>
  
  <p>Maybe I did not use the right search terms, but in any case I'd be happy for a solution</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-15T15:56:17.807Z
Comments.Text: If you want to join the forward and the reverse sequence, why do you want one sequence to be after the other one? Simply it isn't that way in the cell so most programs don't aim to do that. If these are not forward and reverse sequence, did you try perl, python or awk? Are you open to a solution in those languages?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-15T17:02:49.700Z
Comments.Text: Thanks - I added an explanation as second paragraph. These are of course not fw/rev reads. I have unfortunately not used python, perl or awk so far (but am open if it comes with an easy explanation)
Comments.UserDisplayName: Martin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-16T09:01:30.183Z
Comments.Text: From my understanding  of the problem (which might well be wrong): If the sequences are in the same order (i.e. lines correspond to each other) one very simple approach would be just pasting the files together with `paste -d "" File_A.fastq File_B.fastq > target.fastq`. This would of course require some clean up first.
Comments.UserDisplayName: Sebastian Müller
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-20T22:15:06.583Z
Comments.Text: @SebastianMüller: Yes, seq are corresponding and in the same order. And they have the same correspnding names in both files (sorry - I made this more precise in the question), i.e. only the sequence and quality data (i.e .lines 2 + 2n), but not the names (lines 1 + 2n) should be fused. Is there a way to do that, or afterward replace fused names with the original ones by another command? (I am new to Linux, was looking for sth. like this but could not find). If sth. like this works with 10+ mio seq per file, this would be the desired answer!
Comments.UserDisplayName: Martin
Tags: <fastq>
Title: fuse fastq files with multiple records
ThreadText: # fuse fastq files with multiple records
  
  (Asked by: username_0 on 2018-08-15)
  
  I'd like to fuse ~50 pairs of corresponding fastq files *on per sequence bases*. I'd be happy for a solution as a Linux script, R script (if feasible with huge files), or any free software that runs preferably under Debian Linux (otherwise Windows). Aim is preprocessing for a Linux pipeline for NGS data. Ideally both the sequence and quality data are fused.
  
  The **aim** of this request is the following: there is an existing pipeline for downstream mapping of immunoreceptor sequences (migec/mixcr/vdjtools etc.). Previously we got Illumina reads (read 1, read 2 = reverse), with the index and UMI sequences attached to one of the reads => can be alligned and processed to habe 1 biological useful sequence with UMI and index at one end. NOW we get read 1, read 2 and seperately "read 3" which is just the UMI+Index reads, and which was attached by the core facility to read 1 using software we do not have access to. Since the previous pipeline works well, we prefere to use it in the future, but are open for suggestions how to use the index reads with migec/mixcr/vdjtools otherwise.
  
  Each of the 2 files contains the same number (millions) of short (~300 bp) and corresponding DNA reads:
  
  File\_A.fastq:
  
  * @record1 / aaa... / +... / ...
  * @record2 / ...
  * @record3 / ...
  * ... (millions)
  
  File\_B.fastq:
  
  * @record1 / ttt... / +... / ...
  * @record2 / ...
  * @record3 / ...
  * ... (millions)
  
  The target file should look like:
  
  * @record1 / aaa...ttt... / +... / ...
  * @record2 / ...
  * @record3 / ...
  * ... (millions)
  
  (of course the aaa...ttt... is just an example)
  
  I was looking for a solution on Stackexchange and Google, but the solutions, I found do not work for me because
  
  * manual processing is not an option given the number of sequences
  * I need to fuse the files on a **per sequence** basis, i.e. a simple Linux command like cat file1 file2 is not what I need (as far as I understand, this would give me (record1\_A record2\_A.... record1B record2\_B).
  * R, package shortread, readFastq + writeFastq would be perfect, but cannot handle such large numbers of input lines
  
  Maybe I did not use the right search terms, but in any case I'd be happy for a solution
  
  **Tags:** fastq
  
  > **Comments:**
  > **username_1**: If you want to join the forward and the reverse sequence, why do you want one sequence to be after the other one? Simply it isn't that way in the cell so most programs don't aim to do that. If these are not forward and reverse sequence, did you try perl, python or awk? Are you open to a solution in those languages?  
  > **username_0**: Thanks - I added an explanation as second paragraph. These are of course not fw/rev reads. I have unfortunately not used python, perl or awk so far (but am open if it comes with an easy explanation)  
  > **username_2**: From my understanding  of the problem (which might well be wrong): If the sequences are in the same order (i.e. lines correspond to each other) one very simple approach would be just pasting the files together with `paste -d "" File_A.fastq File_B.fastq > target.fastq`. This would of course require some clean up first.  
  > **username_0**: @username_2: Yes, seq are corresponding and in the same order. And they have the same correspnding names in both files (sorry - I made this more precise in the question), i.e. only the sequence and quality data (i.e .lines 2 + 2n), but not the names (lines 1 + 2n) should be fused. Is there a way to do that, or afterward replace fused names with the original ones by another command? (I am new to Linux, was looking for sth. like this but could not find). If sth. like this works with 10+ mio seq per file, this would be the desired answer!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-08-21. Score: 1 | Accepted answer
  
  Based on the comments of the question I thought to put together this answer using [GNU coreutils](https://www.gnu.org/software/coreutils/coreutils.html) which should exist on pretty much all systems.
  
  The idea is to paste the 2 files together with `paste` (assuming they are in the same order and each line correspond to the same line in the other file).
  
  Since only the sequence and quality of both files should be kept, the lines starting with `@` and `+` need to be made empty for one of the files. This can be achived with sed, e.g. `sed 's/^@.*//'` empties all lines starting with `@` but keeps the empty line.
  
  To modify the second file on the fly, one can using [process substitution](http://tldp.org/LDP/abs/html/process-sub.html) `<()`.
  Everything strung together should be something like this:
  
  ```
  file1=File_A.fastq
  file2=File_B.fastq
  paste -d "" $file1 <(cat $file2 | sed 's/^@.*//' | sed 's/^+.*//') > target.fastq
  
  ```
  
  I haven't tested it thoroughly, but could be a good starting point.
  
  If you have a lot of files to paste, you might however want to process them first (not on the fly) using:
  
  ```
  for file in $(ls *.fastq); do 
    cat $file | sed 's/^@.*//' | sed 's/^+.*//' > ${file%.fastq}_cleaned.fastq
  done
  
  ```
  
  This creates `File_A_mod.fastq` from `File_A.fastq` etc.
  Best to move them in a separate directory (make sure the first one is the original instead of the \_cleaned one since it serves as the base).
  You should have the following in the folder: `File_A.fastq File_B_cleaned.fastq File_C_cleaned.fastq ...`.
  Those can be than pasted with `paste -d "" *.fastq > target.fastq`.
  
  
  ---
  
  ### Answer by username_3 on 2018-08-15. Score: 0
  
  What you actually want to do is overlap the reads, for which you can use tools like [FLASH](https://ccb.jhu.edu/software/FLASH/) or [BBMerge](https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/bbmerge-guide/) (I've had better luck with FLASH).
  
  > **Comments:**
  > **username_0**: Thanks, the 2 sequences I want to process are NOT fw/rev or any other overlapping sequences (see new explanation above)  
  > **username_3**: Ah, but at least mixcr will accept the two fastq files as they already are.  
  > **username_0**: Yes, but somehow I need to keep the UMI/index from the 3rd file... If there is a way to do this with  mixcr&co, that would be perfect, but I am not aware of that possibility.  
  > **username_3**: The samples should have been split according to index (I assume this is an Illumina sample index) and the UMI is typically added to the read name (I or someone else can produce a few lines of code to do that for you).  
  > **username_0**: Thanks a lot. Yes, it is Ilumina. Unfortunately sequences are not split for index, and index +UMI are not included in filenames (will add examples as soon as I get faster internet), but Migec can list each index in a barcode file and demultiplex according to index, if supplied with 2 (not 3) files, with UMI+index added to e.g. read 1... (used to work perfectly with this pipeliine as long as we had index+UMI added to read 1)...  
  > **username_3**: Make your life easier and ask your sequencing provider to just give you the files in a standard format (e.g., split by index and with the UMIs in the header). Darn near every tool expects exactly that.  
  
  ---
  
  