Answers.Body: <p><strong>Don't do it:</strong> Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ.  For a detailed description of what can go wrong in FASTQ parsing see for example <a href="http://biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html#FastqGeneralIterator" rel="noreferrer">http://biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html#FastqGeneralIterator</a>.</p>
  
  <p>If the FASTQ is malformed, then you should really ask yourself how this happened in the first place and fix the source of the problem. If the record is valid FASTQ, then I suggest parsing the read with for example <code>FastqGeneralIterator</code> and dumping the parsed result back to FASTQ in a 4-line-per-record form.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-09T09:23:58.110Z
Answers.LastActivityDate: 2018-11-09T09:23:58.110Z
Answers.UserDisplayName: winni2k
Answers.Body: <p>If you are 100% sure the read only has 4 lines (they can have more), you can use this <code>sed</code> command:</p>
  
  <pre><code>sed -i.bak '/^@HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG/,+3d'
  </code></pre>
  
  <p>The <code>-i.bak</code> makes <code>sed</code> modify the original file and create a backup copy with the same name and the extension <code>.bak</code>. The command just means "delete the line matching the pattern and the next three lines".</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T12:44:02.447Z
Answers.Text: Maybe the "@" should be added to the pattern. What would happen if the read name was also on the "+" line?
Answers.UserDisplayName: bli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T09:40:39.210Z
Answers.LastActivityDate: 2018-11-21T19:17:19.677Z
Answers.LastEditDate: 2018-11-21T19:17:19.677Z
Answers.UserDisplayName: terdon
Answers.Body: <p>When I can I like to do file processing line-by-line in the spirit of UNIX tools. You can read 4 lines from a Fastq file into 4 tab-separated values on a single line using <code>paste</code>, and then use <code>grep</code> to filter out the record in question. (Then you just have to turn the tabs back into line breaks.)</p>
  
  <pre><code>paste - - - - &lt; reads.fastq \
      | grep -v 'HWI-D00466:116:CC62WANXX:3:1102:7363:63646' \
      | tr '\t' '\n' \
      &gt; reads-fixed.fastq
  </code></pre>
  
  <p>However, it's likely that other reads in your Fastq file are corrupted as well, in which case it's probably better to write a script in Python or some other language that discards all reads whose read length doesn't match the quality string length.</p>
  
  <p>But of course the most important question is: how did the data become corrupted in the first place? Where did this file come from, and what pre-processing steps were performed on it? Is getting rid of this read (or all reads with mismatched sequence/quality length) going to solve the problem for you? These aren't questions that we can answer for you, but probably deserve good answers. :-)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-01T15:20:17.277Z
Answers.LastActivityDate: 2018-11-01T15:20:17.277Z
Answers.UserDisplayName: Daniel Standage
Answers.Body: <p>Here's an <code>awk</code> one-liner that works as intended, unlike the <code>bioawk</code> answer.</p>
  
  <p>This removes all instances of fastq records in which the length of the seq and qual fields do not match. Again, append <code>| gzip &gt; filtered.fastq.gz</code> to gzip the output.</p>
  
  <pre><code>zcat bad_file.fq.gz | awk '{ pos = NR%4; if ( pos == 1) {h=<span class="math-container">$0} else if (pos == 2) \
    {s=$0} else if (pos==3){c=$0} else if ( pos==0 ) \
    {q=$</span>0; if ( length(q) == length(s) ){printf("%s\n%s\n%s\n%s\n", h, s, c, q)}}}'
  </code></pre>
  
  <p>Delete the <code>\</code> characters at the end of the first two lines to make it easy to paste into your terminal.</p>
  
  <p>Here is a shorter version of the same thing thanks to @terdon</p>
  
  <pre><code>zcat file.fastq.gz | awk '{ record[++k]=<span class="math-container">$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($</span>0) == slen){ for(i in record){print record[i]}} k=0; }}'
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T15:51:28.393Z
Answers.Text: Heh, I was just writing one too :) Here's what I came up with, it's a bit shorter than yours: `zcat file.fastq.gz | awk '{ record[++k]=$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($0) == slen){ for(i in record){print record[i]}} k=0; }}'` feel free to add it to your answer if you want to.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T15:41:27.400Z
Answers.LastActivityDate: 2018-11-01T15:51:00.017Z
Answers.LastEditDate: 2018-11-01T15:51:00.017Z
Answers.UserDisplayName: conchoecia
Answers.Body: <p>To generalize this to remove all reads that do not have the same length of read and sequence quality, you can use this bioawk one-liner:</p>
  
  <pre><code>bioawk -cfastx '{ if (length(<span class="math-container">$seq) == length($</span>qual) ){printf("@%s %s\n%s\n+\n%s\n", <span class="math-container">$name, $</span>comment, <span class="math-container">$seq, $</span>qual)} }' my_fastq.gz
  </code></pre>
  
  <p>This strips anything in the "fastq comment field", <code>+</code>, but that almost never has anything there anyway.</p>
  
  <p>Append <code>| gzip &gt; filtered.fastq.gz</code> to the above command to gzip the output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T14:27:54.283Z
Answers.Text: This doesn't seem to work for me (Arch Linux, bioawk version 20110810). Even after correcting the syntax error (corrected in your answer), it seems to exit after the first record it finds where the lengths are not the same. Can you confirm it works on your machine?
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T15:15:02.677Z
Answers.Text: Thanks for the edit! Yep, I just confirmed that it works for me on all records. I'm using Ubuntu 16.04. `>>bioawk --version` returns `awk version 20110810`. It looks like the last time that I compiled bioawk was from the github repo in 2016.
Answers.UserDisplayName: conchoecia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T15:18:35.250Z
Answers.Text: That's really weird. I have the same version. Can you try a case where the record with different lengths is the second one in the file? Does it not just print the first record there?
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T15:23:56.647Z
Answers.Text: I got the same error as you - I made the second entry in the fastq file have a different length of seq and qual fields and `bioawk` just exited before printing that record. I wonder if this is intended behavior? I have to add that in my above comment, I just tried the bioawk one-liner on a normal fastq file without any length mismatches. Oops!
Answers.UserDisplayName: conchoecia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-31T15:35:30.507Z
Answers.Text: I don't see how it can be intended. Sounds like a bug in bioawk.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-30T19:20:55.403Z
Answers.LastActivityDate: 2018-10-31T13:50:33.817Z
Answers.LastEditDate: 2018-10-31T13:50:33.817Z
Answers.UserDisplayName: conchoecia
Body: <p>I have the following error when running bowtie2:</p>
  
  <blockquote>
    <p>Error: Read HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG
    has more read characters than quality values.</p>
  </blockquote>
  
  <p>I now want to remove all 4 lines of this specific read from the fastq file.</p>
  
  <p>How can I use awk or sed to do this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-23T07:34:01.323Z
Comments.Text: Run `tail` on your FASTQ file. If a job aborted while writing to disk, this read will be the last line. This error message likely means the file is truncated and missing reads. That read is not as much of a concern as the rest of the data. It’s best to run your data processing again to make sure the file is complete.
Comments.UserDisplayName: Tom Kelly ケリー・トム
Tags: <ngs><fastq><filtering><bioawk><text-processing>
Title: Delete all 4 lines of a fastq read from a fastq file using read ID
ThreadText: # Delete all 4 lines of a fastq read from a fastq file using read ID
  
  (Asked by: username_0 on 2018-10-30)
  
  I have the following error when running bowtie2:
  
  > Error: Read HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG
  > has more read characters than quality values.
  
  I now want to remove all 4 lines of this specific read from the fastq file.
  
  How can I use awk or sed to do this?
  
  **Tags:** ngs, fastq, filtering, bioawk, text-processing
  
  > **Comments:**
  > **username_1**: Run `tail` on your FASTQ file. If a job aborted while writing to disk, this read will be the last line. This error message likely means the file is truncated and missing reads. That read is not as much of a concern as the rest of the data. It’s best to run your data processing again to make sure the file is complete.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_2 on 2018-11-09. Score: 9
  
  **Don't do it:** Your FASTQ file is either malformed or your FASTQ record spans more than four lines, which is allowed in FASTQ. For a detailed description of what can go wrong in FASTQ parsing see for example <http://biopython.org/DIST/docs/api/Bio.SeqIO.QualityIO-module.html#FastqGeneralIterator>.
  
  If the FASTQ is malformed, then you should really ask yourself how this happened in the first place and fix the source of the problem. If the record is valid FASTQ, then I suggest parsing the read with for example `FastqGeneralIterator` and dumping the parsed result back to FASTQ in a 4-line-per-record form.
  
  
  ---
  
  ### Answer by username_3 on 2018-10-30. Score: 6
  
  If you are 100% sure the read only has 4 lines (they can have more), you can use this `sed` command:
  
  ```
  sed -i.bak '/^@HWI-D00466:116:CC62WANXX:3:1102:7363:63646 1:N:0:GCACACG/,+3d'
  
  ```
  
  The `-i.bak` makes `sed` modify the original file and create a backup copy with the same name and the extension `.bak`. The command just means "delete the line matching the pattern and the next three lines".
  
  > **Comments:**
  > **username_4**: Maybe the "@" should be added to the pattern. What would happen if the read name was also on the "+" line?  
  
  ---
  
  ### Answer by username_5 on 2018-11-01. Score: 3
  
  When I can I like to do file processing line-by-line in the spirit of UNIX tools. You can read 4 lines from a Fastq file into 4 tab-separated values on a single line using `paste`, and then use `grep` to filter out the record in question. (Then you just have to turn the tabs back into line breaks.)
  
  ```
  paste - - - - < reads.fastq \
      | grep -v 'HWI-D00466:116:CC62WANXX:3:1102:7363:63646' \
      | tr '\t' '\n' \
      > reads-fixed.fastq
  
  ```
  
  However, it's likely that other reads in your Fastq file are corrupted as well, in which case it's probably better to write a script in Python or some other language that discards all reads whose read length doesn't match the quality string length.
  
  But of course the most important question is: how did the data become corrupted in the first place? Where did this file come from, and what pre-processing steps were performed on it? Is getting rid of this read (or all reads with mismatched sequence/quality length) going to solve the problem for you? These aren't questions that we can answer for you, but probably deserve good answers. :-)
  
  
  ---
  
  ### Answer by username_6 on 2018-10-31. Score: 1
  
  Here's an `awk` one-liner that works as intended, unlike the `bioawk` answer.
  
  This removes all instances of fastq records in which the length of the seq and qual fields do not match. Again, append `| gzip > filtered.fastq.gz` to gzip the output.
  
  ```
  zcat bad_file.fq.gz | awk '{ pos = NR%4; if ( pos == 1) {h=$0} else if (pos == 2) \
    {s=$0} else if (pos==3){c=$0} else if ( pos==0 ) \
    {q=$0; if ( length(q) == length(s) ){printf("%s\n%s\n%s\n%s\n", h, s, c, q)}}}'
  
  ```
  
  Delete the `\` characters at the end of the first two lines to make it easy to paste into your terminal.
  
  Here is a shorter version of the same thing thanks to @username_3
  
  ```
  zcat file.fastq.gz | awk '{ record[++k]=$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($0) == slen){ for(i in record){print record[i]}} k=0; }}'
  
  ```
  
  > **Comments:**
  > **username_3**: Heh, I was just writing one too :) Here's what I came up with, it's a bit shorter than yours: `zcat file.fastq.gz | awk '{ record[++k]=$0; if(NR%4==2){slen=length($0)} else if(NR%4==0){if(length($0) == slen){ for(i in record){print record[i]}} k=0; }}'` feel free to add it to your answer if you want to.  
  
  ---
  
  ### Answer by username_6 on 2018-10-30. Score: 0
  
  To generalize this to remove all reads that do not have the same length of read and sequence quality, you can use this bioawk one-liner:
  
  ```
  bioawk -cfastx '{ if (length($seq) == length($qual) ){printf("@%s %s\n%s\n+\n%s\n", $name, $comment, $seq, $qual)} }' my_fastq.gz
  
  ```
  
  This strips anything in the "fastq comment field", `+`, but that almost never has anything there anyway.
  
  Append `| gzip > filtered.fastq.gz` to the above command to gzip the output.
  
  > **Comments:**
  > **username_3**: This doesn't seem to work for me (Arch Linux, bioawk version 20110810). Even after correcting the syntax error (corrected in your answer), it seems to exit after the first record it finds where the lengths are not the same. Can you confirm it works on your machine?  
  > **username_6**: Thanks for the edit! Yep, I just confirmed that it works for me on all records. I'm using Ubuntu 16.04. `>>bioawk --version` returns `awk version 20110810`. It looks like the last time that I compiled bioawk was from the github repo in 2016.  
  > **username_3**: That's really weird. I have the same version. Can you try a case where the record with different lengths is the second one in the file? Does it not just print the first record there?  
  > **username_6**: I got the same error as you - I made the second entry in the fastq file have a different length of seq and qual fields and `bioawk` just exited before printing that record. I wonder if this is intended behavior? I have to add that in my above comment, I just tried the bioawk one-liner on a normal fastq file without any length mismatches. Oops!  
  > **username_3**: I don't see how it can be intended. Sounds like a bug in bioawk.  
  
  ---
  
  