<p>Dear Biostar members,</p>

<p>I am involved in analyzing data from NGS technologies. As you may know, raw data files are in fastq format. And the fact is that these files are huge (several gigabytes each).</p>

<p>As a first step, we naturally want to make some quality controls of the sequencing step. My team and I are aware of some tools like <a href='http://picard.sourceforge.net/'>Picard</a> or <a href='https://github.com/pezmaster31/bamtools/'>bamtools</a> that give some information and statistics, but they do not allow to extract all the information we want. So, at some point, I am afraid we will have to parse fastq files ourselves.</p>

<p>Which programming language would you choose to do the job (<i>ie</i> : read and process a 10GB file) ? Obviously, the main goal is to make the processing as fast as possible.</p>

<ol>
<li>awk/sed</li>
<li>C</li>
<li>Perl</li>
<li>Java</li>
<li>Python</li>
</ol>

<p>Do you think that we will be I/O bound anyway, so using a language or another makes no big difference ? </p>

<p>Would you read the file line by line or load chunks of hundreds of lines (for instance) ?</p>

<p>Many thanks.
T.</p>

<p>The short answer is: it depends a lot on what exactly it is you need to do with the contents of the files.</p>
<p>Regarding being I/O bound, I think it is quite unlikely unless you have the data on very slow disks. Even a single 5400rpm disk can read in the order of 50MB/s, which means that reading a 10GB file will not take much over 3min. Since you are working with large NGS datasets, I would assume that you have a RAID system, in which case the combined speed of multiple disks will almost certainly exceed how fast you can process the data.</p>
<p>When it comes to speed, nothing beats a good C or C++ implementation. However, there is a huge price to pay in terms of development time. If what you do is pretty basic string operations, both Perl and Python should do fine, and the savings in development time are most likely well worth the slightly longer run time compared to a C or C++ implementation.</p>
<p>In my experience, Perl has a more efficient implementation of regular expressions. Conversely, if you need to load all the data into memory, the Python data structures appear to be more memory efficient.</p>
<p>Awk is the only of the languages that you list that I would advise you against. It is very convenient for writing short command-line hacks, but speed is not one of its virtues. I don't have any experience with using Java for bioinformatics tasks.</p>