Answers.Body: <p>I'll add an answer to my own question, which was asked out of lack of experience with cloud (AWS) EC2 service.</p>
  <p>The way I was testing <code>bwa</code> is from a custom AMI image created for our pipelines.</p>
  <p>The image has &quot;preloaded&quot; reference data that we are using for alignement so that we spare some time downloading reference data from <code>S3</code> and spare some <code>S3</code> api calls since we have a lot of files used as reference data.</p>
  <p>Simply our image is single 200GB EBS volume used with <code>AWS ECS AMI</code> where we stored our reference data and created a custom image for Batch out of it. Then we use that AMI for AWS Batch jobs.</p>
  <p>Once I used <code>iotop</code> (after few days of testing) to monitor disk activity I've noticed that <code>bwa</code> loads index into memory very slow, cca <code>5MB/s</code>. Syncing files from S3 was faster then using this <code>snapshot</code> approach.</p>
  <p>For benchmarking I've used the same AMI manually launched. In the AWS documentation there is part which explains why and how you should optimize EBS when AMI is launched from a snapshot.
  <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html</a></p>
  <p>After spining up fresh EC2, syncing reference data and running <code>bwa</code> everything was clear. My EBS volume was slow and had to be initialized first.</p>
  <p>That is all. My lack of EBS storage knowledge was the issue.</p>
  <p>This helped me understand that I should not use custom AMI with ref data stored and will probably use EFS or simply syncing the data from S3 inside Docker entrypoint.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-09T00:24:24.010Z
Answers.LastActivityDate: 2021-08-09T10:36:49.630Z
Answers.LastEditDate: 2021-08-09T10:36:49.630Z
Answers.UserDisplayName: titus
Answers.Body: <p><em>Answer from @user438383, converted from comment:</em></p>
  <p>BWA has slow indexing due to the indexing size; this can be improved by using bwa-mem2:</p>
  <p><a href="https://github.com/bwa-mem2/bwa-mem2" rel="nofollow noreferrer">https://github.com/bwa-mem2/bwa-mem2</a></p>
  <blockquote>
  <p>We are happy to announce that the index size on disk is down by 8 times and in memory by 4 times due to moving to only one type of FM-index (2bit.64 instead of 2bit.64 and 8bit.32) and 8x compression of suffix array. For example, for human genome, index size on disk is down to ~10GB from ~80GB and memory footprint is down to ~10GB from ~40GB. <em><strong>There is a substantial reduction in index IO time due to the reduction</strong></em> and hardly any performance impact on read mapping.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-09T00:57:12.127Z
Answers.Text: Yes but it is not yet production ready. Talking from my very poor amount of reading on the bwa-mem2 topics.
Answers.UserDisplayName: titus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-08T12:31:58.507Z
Answers.LastActivityDate: 2021-08-08T23:25:33.580Z
Answers.LastEditDate: 2021-08-08T23:25:33.580Z
Answers.UserDisplayName: gringer
Body: <p>Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.</p>
  <p>Past week most of work was on testing <code>bwa mem</code>.</p>
  <p>I've noticed that <code>bwa</code> takes about 10 minutes to start processing data. When monitoring <code>bwa</code> resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.</p>
  <p>After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with <code>bwa index</code>.</p>
  <p>When I monitor disk activity with <code>iotop</code> I noticed that <code>disk read</code> is only <code>5MB/s</code>, when I run <code>bwa mem</code> alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple <code>copy</code> test confirms that.</p>
  <p>Is it a &quot;limitation&quot; of <code>bwa</code>, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?</p>
  <p>The same &quot;slow preload&quot; happens when I run <code>bwa shm /path/to/ref</code>.
  Then <code>bwa mem</code> starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-08T22:14:26.180Z
Comments.Text: Issue was with EBS snapshot which holds reference indexed data. Issue is that when you run EC2 from snapshot, EBS storage performance is expected to be low until you optimize the volume. My assumptions and questions for two weeks, all "goes to the water".
  Unfortunately I was not aware of this hence I asked the question out from ignorance.
Comments.UserDisplayName: titus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-08T23:22:58.383Z
Comments.Text: Can you please add this as an actual answer and accept it? This will help the StackExchange algorithms to identify this as a question with and answer, and stop promoting it in the future to fish for more answers.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-09T00:15:58.680Z
Comments.Text: I will. Didn not want to do that imediatelly since it is a little bit awkward to answer own question = )
Comments.UserDisplayName: titus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-09T02:52:37.493Z
Comments.Text: Yeah, I find it awkward as well, but you did already answer the question (as a comment to your own question). Might as well get a bit more credit for that. Also, it's worth thinking about comments as transient things that are there only to help people improve their work; they're not meant to be used for anything substantial that should hang around for a long time (beyond gratitude).
Comments.UserDisplayName: gringer
Tags: <bwa>
Title: Slow bwa shm "preloading" into memory
ThreadText: # Slow bwa shm "preloading" into memory
  
  (Asked by: username_0 on 2021-08-07)
  
  Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.
  
  Past week most of work was on testing `bwa mem`.
  
  I've noticed that `bwa` takes about 10 minutes to start processing data. When monitoring `bwa` resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.
  
  After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with `bwa index`.
  
  When I monitor disk activity with `iotop` I noticed that `disk read` is only `5MB/s`, when I run `bwa mem` alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple `copy` test confirms that.
  
  Is it a "limitation" of `bwa`, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?
  
  The same "slow preload" happens when I run `bwa shm /path/to/ref`.
  Then `bwa mem` starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.
  
  **Tags:** bwa
  
  > **Comments:**
  > **username_0**: Issue was with EBS snapshot which holds reference indexed data. Issue is that when you run EC2 from snapshot, EBS storage performance is expected to be low until you optimize the volume. My assumptions and questions for two weeks, all "goes to the water". Unfortunately I was not aware of this hence I asked the question out from ignorance.  
  > **username_1**: Can you please add this as an actual answer and accept it? This will help the StackExchange algorithms to identify this as a question with and answer, and stop promoting it in the future to fish for more answers.  
  > **username_0**: I will. Didn not want to do that imediatelly since it is a little bit awkward to answer own question = )  
  > **username_1**: Yeah, I find it awkward as well, but you did already answer the question (as a comment to your own question). Might as well get a bit more credit for that. Also, it's worth thinking about comments as transient things that are there only to help people improve their work; they're not meant to be used for anything substantial that should hang around for a long time (beyond gratitude).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2021-08-09. Score: 1 | Accepted answer
  
  I'll add an answer to my own question, which was asked out of lack of experience with cloud (AWS) EC2 service.
  
  The way I was testing `bwa` is from a custom AMI image created for our pipelines.
  
  The image has "preloaded" reference data that we are using for alignement so that we spare some time downloading reference data from `S3` and spare some `S3` api calls since we have a lot of files used as reference data.
  
  Simply our image is single 200GB EBS volume used with `AWS ECS AMI` where we stored our reference data and created a custom image for Batch out of it. Then we use that AMI for AWS Batch jobs.
  
  Once I used `iotop` (after few days of testing) to monitor disk activity I've noticed that `bwa` loads index into memory very slow, cca `5MB/s`. Syncing files from S3 was faster then using this `snapshot` approach.
  
  For benchmarking I've used the same AMI manually launched. In the AWS documentation there is part which explains why and how you should optimize EBS when AMI is launched from a snapshot.
  <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html>
  
  After spining up fresh EC2, syncing reference data and running `bwa` everything was clear. My EBS volume was slow and had to be initialized first.
  
  That is all. My lack of EBS storage knowledge was the issue.
  
  This helped me understand that I should not use custom AMI with ref data stored and will probably use EFS or simply syncing the data from S3 inside Docker entrypoint.
  
  
  ---
  
  ### Answer by username_1 on 2021-08-08. Score: 0
  
  *Answer from @user438383, converted from comment:*
  
  BWA has slow indexing due to the indexing size; this can be improved by using bwa-mem2:
  
  <https://github.com/bwa-mem2/bwa-mem2>
  
  > We are happy to announce that the index size on disk is down by 8 times and in memory by 4 times due to moving to only one type of FM-index (2bit.64 instead of 2bit.64 and 8bit.32) and 8x compression of suffix array. For example, for human genome, index size on disk is down to ~10GB from ~80GB and memory footprint is down to ~10GB from ~40GB. ***There is a substantial reduction in index IO time due to the reduction*** and hardly any performance impact on read mapping.
  
  > **Comments:**
  > **username_0**: Yes but it is not yet production ready. Talking from my very poor amount of reading on the bwa-mem2 topics.  
  
  ---
  
  