<p>Hi everyone</p>

<p>Is there a fast way to do this filter?</p>

<p>I have a huge Fasta file (sequences are short reads coming from an Illumina instrument). I have also a list of nucleotide sequences (not Fasta, just the sequences) and I want to remove from the big Fasta file all entries identical to those in the list.</p>

<p>My idea was simply to go down through the Fasta file and then, for every read, check all the sequences of the list. If the read matches one of the sequences then do nothing, otherwise print the read into a new file. I made this with perl but it takes ages!</p>

<p>The list is made up of nucleotide sequences, not IDs. It's something like this:</p>

<p>AACGACTACTTATCGATC</p>

<p>TCGGCGATATACGTAC</p>

<p>CCAGTTTCGGGGCTAT
....</p>

<p>Thanks!</p>

<ul>
<li>linearize your fastq file into 3 columns: sequence, ID, qualities (using 'awk', or 'perl')</li>
<li>sort it using the sequence as the key ( unix 'sort')</li>
<li>sort your 2nd file of sequences  ( unix 'sort')</li>
<li>use unix '<a href="http://unixhelp.ed.ac.uk/CGI/man-cgi?join">join</a>' to filter out the first file and re-transform it to FASTQ using awk or perl</li>
</ul>
