Answers.Body: <p>Your question can be rewritten as, &quot;how do I get the union of reads in two files?&quot; Since you have already subset the mates, the steps are:</p>
  <ol>
  <li>Process each file, storing the list of reads names in each.</li>
  <li>Take the union of these.</li>
  <li>Iterate through the original/non-subset files and write only the reads with names in the union to the output.</li>
  </ol>
  <p>The files will be both subset correctly and synced properly such that bowtie2 or any other aligner can correctly process them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T12:47:51.937Z
Answers.Text: Hi Devon....is there a tool which can do this ? Kindly let me know.
Answers.UserDisplayName: Amit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T13:40:26.237Z
Answers.Text: Probably, but in the time it would take to search for it it'd be faster to write the 10-20 lines of python (or whatever you prefer) to just do it.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T17:21:44.220Z
Answers.Text: If you want to stick with shell scripting: `comm -12 file1ids file2ds` should give the union, followed by a similar grep like you used but with `-f union.txt` to give the patterns in a file. The undocumented `--no-group-separator` argument to grep may help you too if you're using GNU grep.
Answers.UserDisplayName: Jesse
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T21:38:30.623Z
Answers.Text: Alternately, you could go through both files simultaneously, and output both reads if either of them matches.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T01:38:36.200Z
Answers.Text: Based on my reading of the question, I'd guess they're interested in the *intersection* of reads, rather than the union
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T09:43:20.760Z
Answers.Text: @gringer Could be, I had used the intersection originally in my reply but then figured OP wanted reads with the pattern in either mate. We'd need clarification I guess.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-14T11:02:08.837Z
Answers.LastActivityDate: 2021-01-14T13:41:16.293Z
Answers.LastEditDate: 2021-01-14T13:41:16.293Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>It's a little bit unclear what you're asking, but if you want to repair paired-end files that have become disordered (or the linking has been lost), then the repair function of bbtools should work. This will pair up any coincident read IDs, and create a separate file for any reads that have no pairs in the other file:</p>
  <p><a href="https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/repair-guide/" rel="nofollow noreferrer">https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/repair-guide/</a></p>
  <pre><code>repair.sh in1=broken1.fq in2=broken2 out1=fixed1.fq out2=fixed2.fq outs=singletons.fq repair
  </code></pre>
  <p>If grep is used on one file, and repair used on that file in combination with the original file from the matching read, then the relevant read pairs should appear in the output:</p>
  <pre><code>repair.sh in1=filtered_1.fq in2=file_2 out1=fixed_1.fq out2=fixed_2.fq outs=unpaired_2.fq repair
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-15T05:07:21.160Z
Answers.Text: It's not clear to me that both halves of any pairs will be caught by that grep, so there won't be anything repairable.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T01:35:53.987Z
Answers.Text: That's the "unclear what you're asking" bit.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-15T03:08:16.860Z
Answers.LastActivityDate: 2021-01-17T01:37:24.090Z
Answers.LastEditDate: 2021-01-17T01:37:24.090Z
Answers.UserDisplayName: gringer
Body: <p>I have paired end reads and want to extract only those which have this sequence within them:</p>
  <pre><code>TGTATGTAAACTTCCGACTTCAACTGTA
  </code></pre>
  <p>I tried using <code>grep -B1 -A2 &quot;TGTATGTAAACTTCCGACTTCAACTGTA&quot; file_1.fq | grep -v &quot;^--$&quot;</code>
  and the same from the mate pair. After this I tried aligning them , but they dont align because the reads headers are different.
  Can someone help in extracting the mate pair of one read from the other file (2.fq), so that they can be aligned ?
  regards.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-14T00:55:29.683Z
Comments.Text: Can you sit down and write out **exactly** what you want to do?  I think it's pretty clear why what you did is not returning matched pairs of reads, but it's not clear to me if you see why.
Comments.UserDisplayName: swbarnes2
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-14T08:33:55.697Z
Comments.Text: As I wrote, that I have paired end reads. I want to extract reads from both pairs which have in inserted sequence TGTATGTAAACTTCCGACTTCAACTGTA in them. It belongs to a transposon along with guideDNA. The fastq files are stored as forward and reverse reads. When I executed the script it brings out the reads which have insertion within them. However, they dont align with Bowtie2 as the reads are not from the same pairs.
Comments.UserDisplayName: Amit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-14T21:37:08.167Z
Comments.Text: What I'm trying to get at is, do you want reads which lack the sequence if their mate has it?  Because if the answer is "no", why are you trying to align paired reads when you are throwing away a member of a pair?  And if the answer is "yes", you are clearly throwing those reads away.
Comments.UserDisplayName: swbarnes2
Tags: <grep>
Title: FASTQ file reads extraction paired ends
ThreadText: # FASTQ file reads extraction paired ends
  
  (Asked by: username_0 on 2021-01-13)
  
  I have paired end reads and want to extract only those which have this sequence within them:
  
  ```
  TGTATGTAAACTTCCGACTTCAACTGTA
  
  ```
  
  I tried using `grep -B1 -A2 "TGTATGTAAACTTCCGACTTCAACTGTA" file_1.fq | grep -v "^--$"`
  and the same from the mate pair. After this I tried aligning them , but they dont align because the reads headers are different.
  Can someone help in extracting the mate pair of one read from the other file (2.fq), so that they can be aligned ?
  regards.
  
  **Tags:** grep
  
  > **Comments:**
  > **username_1**: Can you sit down and write out **exactly** what you want to do?  I think it's pretty clear why what you did is not returning matched pairs of reads, but it's not clear to me if you see why.  
  > **username_0**: As I wrote, that I have paired end reads. I want to extract reads from both pairs which have in inserted sequence TGTATGTAAACTTCCGACTTCAACTGTA in them. It belongs to a transposon along with guideDNA. The fastq files are stored as forward and reverse reads. When I executed the script it brings out the reads which have insertion within them. However, they dont align with Bowtie2 as the reads are not from the same pairs.  
  > **username_1**: What I'm trying to get at is, do you want reads which lack the sequence if their mate has it?  Because if the answer is "no", why are you trying to align paired reads when you are throwing away a member of a pair?  And if the answer is "yes", you are clearly throwing those reads away.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-01-14. Score: 3
  
  Your question can be rewritten as, "how do I get the union of reads in two files?" Since you have already subset the mates, the steps are:
  
  1. Process each file, storing the list of reads names in each.
  2. Take the union of these.
  3. Iterate through the original/non-subset files and write only the reads with names in the union to the output.
  
  The files will be both subset correctly and synced properly such that bowtie2 or any other aligner can correctly process them.
  
  > **Comments:**
  > **username_0**: Hi username_2....is there a tool which can do this ? Kindly let me know.  
  > **username_2**: Probably, but in the time it would take to search for it it'd be faster to write the 10-20 lines of python (or whatever you prefer) to just do it.  
  > **username_3**: If you want to stick with shell scripting: `comm -12 file1ids file2ds` should give the union, followed by a similar grep like you used but with `-f union.txt` to give the patterns in a file. The undocumented `--no-group-separator` argument to grep may help you too if you're using GNU grep.  
  > **username_1**: Alternately, you could go through both files simultaneously, and output both reads if either of them matches.  
  > **username_4**: Based on my reading of the question, I'd guess they're interested in the *intersection* of reads, rather than the union  
  > **username_2**: @username_4 Could be, I had used the intersection originally in my reply but then figured OP wanted reads with the pattern in either mate. We'd need clarification I guess.  
  
  ---
  
  ### Answer by username_4 on 2021-01-15. Score: 1
  
  It's a little bit unclear what you're asking, but if you want to repair paired-end files that have become disordered (or the linking has been lost), then the repair function of bbtools should work. This will pair up any coincident read IDs, and create a separate file for any reads that have no pairs in the other file:
  
  <https://jgi.doe.gov/data-and-tools/bbtools/bb-tools-user-guide/repair-guide/>
  
  ```
  repair.sh in1=broken1.fq in2=broken2 out1=fixed1.fq out2=fixed2.fq outs=singletons.fq repair
  
  ```
  
  If grep is used on one file, and repair used on that file in combination with the original file from the matching read, then the relevant read pairs should appear in the output:
  
  ```
  repair.sh in1=filtered_1.fq in2=file_2 out1=fixed_1.fq out2=fixed_2.fq outs=unpaired_2.fq repair
  
  ```
  
  > **Comments:**
  > **username_1**: It's not clear to me that both halves of any pairs will be caught by that grep, so there won't be anything repairable.  
  > **username_4**: That's the "unclear what you're asking" bit.  
  
  ---
  
  