Body: <p>Our lab has been using bcl2fastq v2.20.0.422 to demultiplex RNA-seq data sequenced on an Illumina Novaseq machine on a beefy EC2 instance and we've run into the strange problem: namely that while demultiplexing is very fast, generating the stats files are unbearably slow.</p>
  <p>In a recent example, we demultiplexed one lane of a Novaseq run (total Fastq.gz files were 464 GB) on a m5.24xlarge instance (with 96 vCPUs) with the data being read and written to the same EFS mount. The demultiplexing took only ~70 minutes, while generating the Stats files took over 4 hours (14,557 seconds to be exact)!</p>
  <p>Running with log level INFO, the demultiplexing progressed quickly as expected, all Fastq files were generated, and the bcl2fastq output the message:</p>
  <pre><code>INFO: Created InterOp file '&quot;./InterOp/IndexMetricsOut.bin&quot;
  </code></pre>
  <p>and sat for hours using &lt;1% CPU writing to <code>IndexMetricsOut.bin</code> <em>very slowly</em>. Tracking the size of the <code>IndexMetricsOut.bin</code> file showed that it was being continuously written to at the rate of about 4kb per second. It did eventually finish (final size 49M) and bcl2fastq reported that it completed successfully, but it seems crazy to me that this stage could take so much longer than actually demultiplexing almost half a TB of data.</p>
  <p>Does anyone have any ideas <em>why</em> the stats files are taking so long to generate, and what I could do about it? Additionally, since the fastq files seem to be completed, is it safe to just move on to alignment without waiting for bcl2fastq to finish?</p>
  <p>NOTE: this question was crossposted on Biostars: <a href="http://biostars.org/p/9491612" rel="nofollow noreferrer">biostars.org/p/9491612</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-29T17:25:18.657Z
Comments.Text: cross-posted:https://www.biostars.org/p/9491612/
Comments.UserDisplayName: swbarnes2
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-29T17:37:53.310Z
Comments.Text: @swbarnes2 That's not against the rules, is it? Since this site is still pretty small, I figured crossposting would increase the odds that someone would be able to answer it
Comments.UserDisplayName: divibisan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-29T18:36:24.270Z
Comments.Text: @divibisan it's not against the rules, but it's considered polite to at least make it clear your crossposted it in both posts, otherwise it runs the risk of people wasting their time answering a question which has already been answered elsewhere.
Comments.UserDisplayName: user438383
Tags: <rna-seq><ngs><illumina><demultiplexing><aws>
Title: Why is bcl2fastq2 taking so long to calculate stats?
ThreadText: # Why is bcl2fastq2 taking so long to calculate stats?
  
  (Asked by: username_0 on 2021-09-29)
  
  Our lab has been using bcl2fastq v2.20.0.422 to demultiplex RNA-seq data sequenced on an Illumina Novaseq machine on a beefy EC2 instance and we've run into the strange problem: namely that while demultiplexing is very fast, generating the stats files are unbearably slow.
  
  In a recent example, we demultiplexed one lane of a Novaseq run (total Fastq.gz files were 464 GB) on a m5.24xlarge instance (with 96 vCPUs) with the data being read and written to the same EFS mount. The demultiplexing took only ~70 minutes, while generating the Stats files took over 4 hours (14,557 seconds to be exact)!
  
  Running with log level INFO, the demultiplexing progressed quickly as expected, all Fastq files were generated, and the bcl2fastq output the message:
  
  ```
  INFO: Created InterOp file '"./InterOp/IndexMetricsOut.bin"
  
  ```
  
  and sat for hours using <1% CPU writing to `IndexMetricsOut.bin` *very slowly*. Tracking the size of the `IndexMetricsOut.bin` file showed that it was being continuously written to at the rate of about 4kb per second. It did eventually finish (final size 49M) and bcl2fastq reported that it completed successfully, but it seems crazy to me that this stage could take so much longer than actually demultiplexing almost half a TB of data.
  
  Does anyone have any ideas *why* the stats files are taking so long to generate, and what I could do about it? Additionally, since the fastq files seem to be completed, is it safe to just move on to alignment without waiting for bcl2fastq to finish?
  
  NOTE: this question was crossposted on Biostars: [biostars.org/p/9491612](http://biostars.org/p/9491612)
  
  **Tags:** rna-seq, ngs, illumina, demultiplexing, aws
  
  > **Comments:**
  > **username_1**: cross-posted:https://www.biostars.org/p/9491612/  
  > **username_0**: @username_1 That's not against the rules, is it? Since this site is still pretty small, I figured crossposting would increase the odds that someone would be able to answer it  
  > **username_2**: @username_0 it's not against the rules, but it's considered polite to at least make it clear your crossposted it in both posts, otherwise it runs the risk of people wasting their time answering a question which has already been answered elsewhere.  
  
  ---
  
  ## 0 answers
  
  