Answers.Body: <p>I usually use <code>samtools sort -@4 -m4g</code>. Samtools will be faster with more threads and more RAM, but I <em>feel</em> the gain is not significant. I haven't done a proper evaluation, but I have run samtools sort with different settings from time to time on clusters when I have to consider speed-resource tradeoffs.</p>
  <p>When you use samtools, make sure you compile it with <a href="https://github.com/ebiggers/libdeflate" rel="noreferrer">libdeflate</a>, which is about twice as fast as zlib on compression. I don't know if the bioconda version built with libdeflate. You may consider the precompiled binary <a href="https://zenodo.org/record/5731013" rel="noreferrer">here</a>.</p>
  <p>Samtools sort, especially with libdeflate, is fairly fast these days. Performance metrics in the other answer are outdated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T16:38:17.883Z
Answers.Text: Super helpful info about `libdeflate`. I was not aware of that. I really want to do a comparison, but it's difficult to make the time. I may do it soon, and will try to include a comparison between `zlib` and `libdeflate`.
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-12T16:48:26.357Z
Answers.Text: Side note: if you want to see if the samtools binary you're using is using `zlib` or `libdeflate`, you can see which libraries it was compiled against using `ldd /path/to/<samtools_binary>`. `zlib` will show as something similar to `libz.so.1`.
Answers.UserDisplayName: Mark Ebbert
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-11T20:02:38.620Z
Answers.LastActivityDate: 2022-02-11T20:08:39.273Z
Answers.LastEditDate: 2022-02-11T20:08:39.273Z
Answers.UserDisplayName: user172818
Answers.Body: <h1>Update 2</h1>
  <p>Ok, for what I expect will be my final update for this answer, I compared the following:</p>
  <ol>
  <li><code>samtools 1.14</code> + <code>zlib</code></li>
  <li><code>samtools 1.15</code> + <code>libdeflate</code> (this is a different version, but shouldn't have a major effect. They released a new version just before I asked cluster admins to compile against <code>libdeflate</code> and didn't notice until later.)</li>
  <li><code>sambamba v0.8.2</code></li>
  </ol>
  <p><strong>Note</strong>: This time I used the following settings:</p>
  <ol>
  <li>Only two samples instead of three</li>
  <li>Max <code>Nextflow</code> queue size of 30 to avoid too many threads reading from the same two files.</li>
  <li>I tested CPU requests from 1 to 7 (step size 1) and then from 9 to 17 (step size 2).</li>
  <li>For jobs where I allotted few CPUs and low memory, I provided a 200% buffer to prevent the jobs from failing with <code>OUT OF MEMORY</code> errors. i.e., I gave the <code>SLURM</code> job 2x the amount of memory than I told the tools they could use. <code>sambamba</code> was the worst offender for exceeding memory in these conditions, but I gave them all 200% buffer to keep comparisons equal.</li>
  </ol>
  <p>Seems pretty clear to me that <code>sambamba</code> is the fastest and uses requested CPU resources most efficiently (i.e., is able to use all of the CPUs requested up until ~8 or 9 CPUs). For reference, <strong>the difference between <code>sambamba</code> and <code>samtools + libdeflate</code> with 9 CPUs is ~10 minutes on average.</strong></p>
  <p>We will certainly be using <code>sambamba</code> going forward, probably with ~9 CPUs and ~9GB per thread since I had to provide a 200% buffer in lower CPU/Mem jobs to avoid job failure, anyway.</p>
  <p>Again, all code and results are available on my GitHub repository (<a href="https://github.com/mebbert/Samtools_sort_optimization_test" rel="noreferrer">Samtools sort optimization test</a>).</p>
  <h2>(Angle 1) Comparison between <code>samtools + zlib</code>, <code>samtools + libdeflate</code> (Lsamtools), and <code>sambamba</code></h2>
  <p>Here, I'm showing the general trend that <code>real time</code> decreases with increased CPUs for all three tools, but Mem Per CPU doesn't have a huge effect. <code>sambamba</code> out performs <code>samtools + zlib</code> regardless of settings. <code>samtools + libdeflate</code> out performs <code>samtools + zlib</code> until ~11 CPUs, where they perform the same (plot 3 below explains why).</p>
  <p><strong>Interesting finding:</strong> <code>samtools + libdeflate</code> performs better than <code>sambamba</code> with a single thread, but <code>sambamba</code> takes over from there as CPUs increase (again, plot #3 explains why).</p>
  <p><a href="https://i.sstatic.net/87BoN.png" rel="noreferrer"><img src="https://i.sstatic.net/87BoN.png" alt="samtools, Lsamtools, and sambamba" /></a></p>
  <h2>(Angle 2) Comparison between <code>samtools + zlib</code>, <code>samtools + libdeflate</code> (Lsamtools), and <code>sambamba</code></h2>
  <p>Here, I'm showing that <code>samtools + zlib</code> and <code>sambamba</code> performance flattens out at ~11 CPUs (again, plot 3 below explans why).</p>
  <p><a href="https://i.sstatic.net/B8WfV.png" rel="noreferrer"><img src="https://i.sstatic.net/B8WfV.png" alt="samtools, Lsamtools, and sambamba" /></a></p>
  <h2>Requested CPUs vs. CPU utilization for <code>samtools + zlib</code>, <code>samtools + libdeflate</code> (Lsamtools), and <code>sambamba</code></h2>
  <p>Here, I'm showing how well each tool is able to utilize the CPUs allotted to it. It was fascinating to see that <code>samtools + libdeflate</code> flattens out very early in it's CPU utilization (i.e., ~3 CPUS) and struggles to cross the 400% CPU utilization with even &gt;10 CPUs. This is why <code>samtools + libdeflate</code> performance flattens out quickly.</p>
  <p><code>samtools + zlib</code> does a much better job utilizing allotted CPUs, but still lags behind <code>sambamba</code> throughout (and is slower, anyway).
  <a href="https://i.sstatic.net/eiSsn.png" rel="noreferrer"><img src="https://i.sstatic.net/eiSsn.png" alt="CPU usage by requested CPUs" /></a></p>
  <h1>Update 1</h1>
  <p><a href="https://i.sstatic.net/jTNaG.png" rel="noreferrer"><img src="https://i.sstatic.net/jTNaG.png" alt="Realtime vs. CPUs and mem per thread (samtools sort)" /></a></p>
  <p>My first plots (below) were showing <code>duration</code> on the <code>z-axis</code>, which is the time from job <em>submission</em> to <em>completion</em> rather than job <em>start</em> to <em>completion</em>. That's why the larger memory requests were much longer in the original plots. <code>Duration</code> is still a useful measure, but I'm now including <code>real time</code>, which is from <em>start</em> to <em>completion</em>.</p>
  <p>Judging solely by <code>real time</code> (above), there is a clear gain between 5 &amp; 7 CPUs (~10 minutes, on average when compared to the same mem-per-thread).</p>
  <p>More data coming, including <code>samtools</code> compiled against <code>libdeflate</code> and <code>sambamba</code>.</p>
  <hr />
  <h1>Original post</h1>
  <p>I made time to semi-formally look at this. I'm glad I did because the results surprised me. I don't answer all of my questions directly, but these results are good enough to get me going in the right direction.</p>
  <p>My code and results are in my <code>GitHub</code> repository (<a href="https://github.com/mebbert/Samtools_sort_optimization_test" rel="noreferrer">Samtools_sort_optimization_test</a>) where you will also find an interactive plot you can download to see the results more clearly (see <code>SAMTOOLS_SORT/samtools_sort_CPU_and_memory_comparison.tar.gz</code> in the repository). It's hard to truly appreciate the 3D <code>plotly</code> plot until you start rotating it.</p>
  <p><strong>NOTES:</strong></p>
  <ol>
  <li>Even though this is from only three samples, I think my results are accurate. I welcome any feedback in case there's a bug.</li>
  <li>In my experience, <code>samtools</code> often exceeds its allotted memory (at least on a <code>SLURM</code> scheduler). i.e., the job gets killed by <code>SLURM</code>. To avoid this, I requested an additional 20% of memory to provide a buffer for <code>samtools</code>. I don't know if this discrepancy is because of <code>SLURM</code> or <code>samtools</code>, though.</li>
  <li>These results are using <code>samtools</code> compiled against <code>zlib</code>. We are preparing a version of <code>samtools</code> compiled against <code>libdeflate</code>, per @user172818's suggestion. I will add those results once completed. I may also compare against <code>sambamba</code>, per @acvill's suggestion.</li>
  <li>These results are related to my other question about <a href="https://bioinformatics.stackexchange.com/questions/18539/samtools-sort-most-efficient-approach-to-sort-a-single-sample-after-aligning-sp">the best sorting/merging method</a>. I will post those results in that question and provide a brief summary here.</li>
  </ol>
  <h1>tl;dr (too long, didn't read)</h1>
  <h2>Primary conclusions based on the plots below</h2>
  <ol>
  <li><strong>Optimal CPUs</strong> (results as expected): There's a significant gain with some additional CPUs, but it levels off around 7 CPUs (total, not 'additional'; i.e., <code>samtools sort -@ 6</code>). I thought we might still see <em>some</em> nice gains above 7, but this is probably what most would expect (e.g., similar to what @user172818 suggested).</li>
  <li><strong>Optimal memory per thread</strong> (<s>results NOT as expected</s>): <s>There is very little to gain with more than 1GB of memory (per thread); that surprised me. There are <em>some</em> performance gains up until ~7GB per thread (~2-4 minutes faster). What really surprised me, however, is that performance dramatically <strong>decreases</strong> as memory per thread goes &gt;7GB</s>. The updated plot (above) shows there is some gain with &gt;1GB mem per thread. When running lots of samples at once (hundreds to 1000), mem per thread seems to play a much larger role (data coming).</li>
  <li><strong>Effect from sorting status</strong> (results as expected): Largely in an attempt to do this as quickly as possible (laziness?), I used three samples, where one was from an aligned and unsorted <code>.bam</code> file, and the other two were from <code>.bam</code> files that had been aligned and sorted, but I then collated them (<code>samtools collate</code>), rather than re-aligning them. Unsurprisingly, the collated samples (not the average use case) sorted slightly faster than the
  unsorted sample (average use case).</li>
  </ol>
  <h2>Initial opinion</h2>
  <ol>
  <li><strong>&quot;Small&quot; sample set</strong>: For a small sample set, I would probably use 7 total CPUs and 7GB per thread.</li>
  <li><strong>&quot;Large&quot; sample set</strong>: I will probably use 7 CPUs (maybe 5) with only 1GB memory per thread. That will allow the scheduler to keep as many jobs going as possible.</li>
  </ol>
  <h2><code>samtools sort</code>: duration vs. CPUs and mem per thread</h2>
  <p>As mentioned above, an interactive version of this plot is available on my <code>GitHub</code>.</p>
  <p><strong>Angle 1</strong>
  <a href="https://i.sstatic.net/o79Ek.png" rel="noreferrer"><img src="https://i.sstatic.net/o79Ek.png" alt="Duration vs. CPUs and mem per thread (samtools sort)" /></a></p>
  <p><strong>Angle 2</strong>
  <a href="https://i.sstatic.net/UGNSu.png" rel="noreferrer"><img src="https://i.sstatic.net/UGNSu.png" alt="Duration vs. CPUs and mem per thread (samtools sort)" /></a></p>
  <h1>Additional details</h1>
  <p>Analyses were run on <a href="https://docs.ccs.uky.edu/display/HPC/MCC+system+overview" rel="noreferrer">University of Kentucky's MCC Cluster</a>, which uses <code>AMD 7702P</code> processors, using <code>samtools v.1.14</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-19T19:45:05.990Z
Answers.LastActivityDate: 2022-02-27T01:12:53.420Z
Answers.LastEditDate: 2022-02-27T01:12:53.420Z
Answers.UserDisplayName: Mark Ebbert
Body: <p>We're preparing to analyze thousands of <code>.bam</code> files beginning with re-alignment, sorting, etc.</p>
  <p><strong>Complicated question:</strong> Has anyone investigated the optimal thread and memory settings to sort hundreds to thousands of <code>.bam</code> files using <code>samtools sort</code> on a cluster?</p>
  <p>By 'optimal', I'm looking to sort as many samples as possible in the shortest time frame. I'm mostly interested in optimizing <code>samtools sort</code> because the other steps (e.g., alignment) are much more obvious to fine-tune.</p>
  <p>For a single sample, throwing large amounts of memory + threads is trivial, but that isn't scalable for thousands of samples. There is also likely a <a href="https://en.wikipedia.org/wiki/Diminishing_returns" rel="nofollow noreferrer">diminishing return</a> at a certain point, even for a single sample.</p>
  <p>Ideally, I'd like to see plots identifying the 'sweet spot' for threads + memory for a single sample (e.g., x == threads, y == memory, z == time), and then plots maximizing throughput for more samples. I would expect the data to follow a -log function, where we could identify the optimal settings at the <a href="https://en.wikipedia.org/wiki/Knee_of_a_curve#:%7E:text=In%20mathematics%2C%20a%20knee%20of,or%20in%20the%20other%20direction." rel="nofollow noreferrer">knee in the curve</a>.</p>
  <p>There are obviously many other factors (e.g., cluster size, administrative scheduler settings, etc.), but I suspect findings on a given cluster will generalize to other clusters.</p>
  <p>Really appreciate any feedback.</p>
  
Tags: <phylogenetics><sequence-analysis><samtools><benchmarking>
Title: Samtools sort: most efficient memory and thread settings for many samples on a cluster
ThreadText: # Samtools sort: most efficient memory and thread settings for many samples on a cluster
  
  (Asked by: username_0 on 2022-02-10)
  
  We're preparing to analyze thousands of `.bam` files beginning with re-alignment, sorting, etc.
  
  **Complicated question:** Has anyone investigated the optimal thread and memory settings to sort hundreds to thousands of `.bam` files using `samtools sort` on a cluster?
  
  By 'optimal', I'm looking to sort as many samples as possible in the shortest time frame. I'm mostly interested in optimizing `samtools sort` because the other steps (e.g., alignment) are much more obvious to fine-tune.
  
  For a single sample, throwing large amounts of memory + threads is trivial, but that isn't scalable for thousands of samples. There is also likely a [diminishing return](https://en.wikipedia.org/wiki/Diminishing_returns) at a certain point, even for a single sample.
  
  Ideally, I'd like to see plots identifying the 'sweet spot' for threads + memory for a single sample (e.g., x == threads, y == memory, z == time), and then plots maximizing throughput for more samples. I would expect the data to follow a -log function, where we could identify the optimal settings at the [knee in the curve](https://en.wikipedia.org/wiki/Knee_of_a_curve#:%7E:text=In%20mathematics%2C%20a%20knee%20of,or%20in%20the%20other%20direction.).
  
  There are obviously many other factors (e.g., cluster size, administrative scheduler settings, etc.), but I suspect findings on a given cluster will generalize to other clusters.
  
  Really appreciate any feedback.
  
  **Tags:** phylogenetics, sequence-analysis, samtools, benchmarking
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-02-19. Score: 8 | Accepted answer
  
  # Update 2
  
  Ok, for what I expect will be my final update for this answer, I compared the following:
  
  1. `samtools 1.14` + `zlib`
  2. `samtools 1.15` + `libdeflate` (this is a different version, but shouldn't have a major effect. They released a new version just before I asked cluster admins to compile against `libdeflate` and didn't notice until later.)
  3. `sambamba v0.8.2`
  
  **Note**: This time I used the following settings:
  
  1. Only two samples instead of three
  2. Max `Nextflow` queue size of 30 to avoid too many threads reading from the same two files.
  3. I tested CPU requests from 1 to 7 (step size 1) and then from 9 to 17 (step size 2).
  4. For jobs where I allotted few CPUs and low memory, I provided a 200% buffer to prevent the jobs from failing with `OUT OF MEMORY` errors. i.e., I gave the `SLURM` job 2x the amount of memory than I told the tools they could use. `sambamba` was the worst offender for exceeding memory in these conditions, but I gave them all 200% buffer to keep comparisons equal.
  
  Seems pretty clear to me that `sambamba` is the fastest and uses requested CPU resources most efficiently (i.e., is able to use all of the CPUs requested up until ~8 or 9 CPUs). For reference, **the difference between `sambamba` and `samtools + libdeflate` with 9 CPUs is ~10 minutes on average.**
  
  We will certainly be using `sambamba` going forward, probably with ~9 CPUs and ~9GB per thread since I had to provide a 200% buffer in lower CPU/Mem jobs to avoid job failure, anyway.
  
  Again, all code and results are available on my GitHub repository ([Samtools sort optimization test](https://github.com/mebbert/Samtools_sort_optimization_test)).
  
  ## (Angle 1) Comparison between `samtools + zlib`, `samtools + libdeflate` (Lsamtools), and `sambamba`
  
  Here, I'm showing the general trend that `real time` decreases with increased CPUs for all three tools, but Mem Per CPU doesn't have a huge effect. `sambamba` out performs `samtools + zlib` regardless of settings. `samtools + libdeflate` out performs `samtools + zlib` until ~11 CPUs, where they perform the same (plot 3 below explains why).
  
  **Interesting finding:** `samtools + libdeflate` performs better than `sambamba` with a single thread, but `sambamba` takes over from there as CPUs increase (again, plot #3 explains why).
  
  [![samtools, Lsamtools, and sambamba](https://i.sstatic.net/87BoN.png)](https://i.sstatic.net/87BoN.png)
  
  ## (Angle 2) Comparison between `samtools + zlib`, `samtools + libdeflate` (Lsamtools), and `sambamba`
  
  Here, I'm showing that `samtools + zlib` and `sambamba` performance flattens out at ~11 CPUs (again, plot 3 below explans why).
  
  [![samtools, Lsamtools, and sambamba](https://i.sstatic.net/B8WfV.png)](https://i.sstatic.net/B8WfV.png)
  
  ## Requested CPUs vs. CPU utilization for `samtools + zlib`, `samtools + libdeflate` (Lsamtools), and `sambamba`
  
  Here, I'm showing how well each tool is able to utilize the CPUs allotted to it. It was fascinating to see that `samtools + libdeflate` flattens out very early in it's CPU utilization (i.e., ~3 CPUS) and struggles to cross the 400% CPU utilization with even >10 CPUs. This is why `samtools + libdeflate` performance flattens out quickly.
  
  `samtools + zlib` does a much better job utilizing allotted CPUs, but still lags behind `sambamba` throughout (and is slower, anyway).
  [![CPU usage by requested CPUs](https://i.sstatic.net/eiSsn.png)](https://i.sstatic.net/eiSsn.png)
  
  # Update 1
  
  [![Realtime vs. CPUs and mem per thread (samtools sort)](https://i.sstatic.net/jTNaG.png)](https://i.sstatic.net/jTNaG.png)
  
  My first plots (below) were showing `duration` on the `z-axis`, which is the time from job *submission* to *completion* rather than job *start* to *completion*. That's why the larger memory requests were much longer in the original plots. `Duration` is still a useful measure, but I'm now including `real time`, which is from *start* to *completion*.
  
  Judging solely by `real time` (above), there is a clear gain between 5 & 7 CPUs (~10 minutes, on average when compared to the same mem-per-thread).
  
  More data coming, including `samtools` compiled against `libdeflate` and `sambamba`.
  
  ---
  
  # Original post
  
  I made time to semi-formally look at this. I'm glad I did because the results surprised me. I don't answer all of my questions directly, but these results are good enough to get me going in the right direction.
  
  My code and results are in my `GitHub` repository ([Samtools\_sort\_optimization\_test](https://github.com/mebbert/Samtools_sort_optimization_test)) where you will also find an interactive plot you can download to see the results more clearly (see `SAMTOOLS_SORT/samtools_sort_CPU_and_memory_comparison.tar.gz` in the repository). It's hard to truly appreciate the 3D `plotly` plot until you start rotating it.
  
  **NOTES:**
  
  1. Even though this is from only three samples, I think my results are accurate. I welcome any feedback in case there's a bug.
  2. In my experience, `samtools` often exceeds its allotted memory (at least on a `SLURM` scheduler). i.e., the job gets killed by `SLURM`. To avoid this, I requested an additional 20% of memory to provide a buffer for `samtools`. I don't know if this discrepancy is because of `SLURM` or `samtools`, though.
  3. These results are using `samtools` compiled against `zlib`. We are preparing a version of `samtools` compiled against `libdeflate`, per @username_1's suggestion. I will add those results once completed. I may also compare against `sambamba`, per @acvill's suggestion.
  4. These results are related to my other question about [the best sorting/merging method](https://bioinformatics.stackexchange.com/questions/18539/samtools-sort-most-efficient-approach-to-sort-a-single-sample-after-aligning-sp). I will post those results in that question and provide a brief summary here.
  
  # tl;dr (too long, didn't read)
  
  ## Primary conclusions based on the plots below
  
  1. **Optimal CPUs** (results as expected): There's a significant gain with some additional CPUs, but it levels off around 7 CPUs (total, not 'additional'; i.e., `samtools sort -@ 6`). I thought we might still see *some* nice gains above 7, but this is probably what most would expect (e.g., similar to what @username_1 suggested).
  2. **Optimal memory per thread** (~~results NOT as expected~~): ~~There is very little to gain with more than 1GB of memory (per thread); that surprised me. There are *some* performance gains up until ~7GB per thread (~2-4 minutes faster). What really surprised me, however, is that performance dramatically **decreases** as memory per thread goes >7GB~~. The updated plot (above) shows there is some gain with >1GB mem per thread. When running lots of samples at once (hundreds to 1000), mem per thread seems to play a much larger role (data coming).
  3. **Effect from sorting status** (results as expected): Largely in an attempt to do this as quickly as possible (laziness?), I used three samples, where one was from an aligned and unsorted `.bam` file, and the other two were from `.bam` files that had been aligned and sorted, but I then collated them (`samtools collate`), rather than re-aligning them. Unsurprisingly, the collated samples (not the average use case) sorted slightly faster than the
     unsorted sample (average use case).
  
  ## Initial opinion
  
  1. **"Small" sample set**: For a small sample set, I would probably use 7 total CPUs and 7GB per thread.
  2. **"Large" sample set**: I will probably use 7 CPUs (maybe 5) with only 1GB memory per thread. That will allow the scheduler to keep as many jobs going as possible.
  
  ## `samtools sort`: duration vs. CPUs and mem per thread
  
  As mentioned above, an interactive version of this plot is available on my `GitHub`.
  
  **Angle 1**
  [![Duration vs. CPUs and mem per thread (samtools sort)](https://i.sstatic.net/o79Ek.png)](https://i.sstatic.net/o79Ek.png)
  
  **Angle 2**
  [![Duration vs. CPUs and mem per thread (samtools sort)](https://i.sstatic.net/UGNSu.png)](https://i.sstatic.net/UGNSu.png)
  
  # Additional details
  
  Analyses were run on [University of Kentucky's MCC Cluster](https://docs.ccs.uky.edu/display/HPC/MCC+system+overview), which uses `AMD 7702P` processors, using `samtools v.1.14`
  
  
  ---
  
  ### Answer by username_1 on 2022-02-11. Score: 8
  
  I usually use `samtools sort -@4 -m4g`. Samtools will be faster with more threads and more RAM, but I *feel* the gain is not significant. I haven't done a proper evaluation, but I have run samtools sort with different settings from time to time on clusters when I have to consider speed-resource tradeoffs.
  
  When you use samtools, make sure you compile it with [libdeflate](https://github.com/ebiggers/libdeflate), which is about twice as fast as zlib on compression. I don't know if the bioconda version built with libdeflate. You may consider the precompiled binary [here](https://zenodo.org/record/5731013).
  
  Samtools sort, especially with libdeflate, is fairly fast these days. Performance metrics in the other answer are outdated.
  
  > **Comments:**
  > **username_0**: Super helpful info about `libdeflate`. I was not aware of that. I really want to do a comparison, but it's difficult to make the time. I may do it soon, and will try to include a comparison between `zlib` and `libdeflate`.  
  > **username_0**: Side note: if you want to see if the samtools binary you're using is using `zlib` or `libdeflate`, you can see which libraries it was compiled against using `ldd /path/to/<samtools_binary>`. `zlib` will show as something similar to `libz.so.1`.  
  
  ---
  
  