<p>Hello BioStar,</p>

<p>After some time working with Illumina and pipelines, I've identified a bottleneck when getting early "draft" results from a run. Figuring out how the sequencing data looks like <strong>in real time</strong>, as opposed to wait for the run to finish after ~11 days. </p>

<p>The goal would be to get an estimate of how many reads one could expect to get for each sample, thereby guiding setup for a subsequent run for topping up the data for those samples that do not reach the required amounts. It would be help a lot if we could reduce the the wall-clock time for reaching a decision on which samples need to be re-run.</p>

<p>When it comes to implementation, I've been thinking on a file status daemon such as Guard[1], coupled with the CASAVA/OLB tools from illumina, performing basecalling and demultiplexing <strong>as soon as the files get written to disk</strong>, without having to wait for the whole run to finish.</p>

<p>Other more high-tech solutions would involve Hadoop Seal[2] and Flume[3] to do the streaming part into Hadoop... but before digging more into this issue I wondered what are you guys doing to get a <strong>draft view on a running sequencing run</strong>.</p>

<p>Cheers &amp; happy new year Bio* !</p>

<p>[1] <a href="https://github.com/guard/guard">https://github.com/guard/guard</a></p>

<p>[2] <a href="http://biodoop-seal.sourceforge.net/installation_generic.html">http://biodoop-seal.sourceforge.net/installation_generic.html</a></p>

<p>[3] <a href="http://www.slideshare.net/cloudera/inside-flume">http://www.slideshare.net/cloudera/inside-flume</a></p>

<p>We're waiting for the whole run to finish before getting any QC information.  Optimizing that step like you say would surely be beneficial, and if you can get the base calling on the index read to run as soon as it's been sequenced, then I think the rest would be simple.</p>
<p>Since you only want to get an estimate of the read counts for each of the multiplexed sample, I would try simply using a script to calculate the index read frequencies.  You could even exclude reads that have quality problems from your counts.  I don't think performance would be an issue since you'd only be dealing with the index reads for one run but, if necessary, you could quickly copy the index reads to a Hadoop cluster and run a custom Pydoop script (or even recycle the Hadoop workcount example) to generate a table (index read, count), which you could join to your sample sheet to get your sample read counts.</p>
<p>HTH.</p>