Answers.Body: <p>Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency.  So <code>samtools merge</code> (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), <code>picard MergeSamFiles</code>, etc.</p>
  
  <p>These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors.  Use <code>ulimit</code> to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc.  Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T07:23:06.240Z
Answers.Text: Maybe merging stepwise as you mention in your second paragraph could speed up things as well as one doesn't hit IO limitations?
Answers.UserDisplayName: sargas
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T07:26:34.097Z
Answers.Text: No, some of the data is now being written and read more than once, so it must be slower.  With crafty use of pipes and process substitution (`samtools merge … <(samtools merge …) …`) this can be minimised and avoid hitting disk, so the slowdown can be trivial (at the cost of using a few more cores).
Answers.UserDisplayName: John Marshall
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T07:18:49.330Z
Answers.LastActivityDate: 2017-05-19T07:18:49.330Z
Answers.UserDisplayName: John Marshall
Answers.Body: <p><code>samtools merge merged.bam *.bam</code> is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T07:18:44.133Z
Answers.Text: Any idea of how this compares to bamtools merge?
Answers.UserDisplayName: Scott Gigante
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T07:20:45.177Z
Answers.Text: Samtools tends to be a bit slower than other tools since it's the reference implementation, so I'd expect bamtools to be a bit faster.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T11:29:01.210Z
Answers.Text: Bamtools is usually slower because it is not optimized and now unmaintained. Sambamba etc are developed with a specific goal to be faster.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-19T07:06:35.520Z
Answers.LastActivityDate: 2017-05-19T07:06:35.520Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.</p>
  
  <p>I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?</p>
  
Tags: <phylogenetics><sam><samtools>
Title: Merge hundreds of small BAM files into a single BAM file
ThreadText: # Merge hundreds of small BAM files into a single BAM file
  
  (Asked by: username_0 on 2017-05-19)
  
  I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.
  
  I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?
  
  **Tags:** phylogenetics, sam, samtools
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2017-05-19. Score: 15 | Accepted answer
  
  `samtools merge merged.bam *.bam` is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.
  
  > **Comments:**
  > **username_0**: Any idea of how this compares to bamtools merge?  
  > **username_3**: Samtools tends to be a bit slower than other tools since it's the reference implementation, so I'd expect bamtools to be a bit faster.  
  > **username_4**: Bamtools is usually slower because it is not optimized and now unmaintained. Sambamba etc are developed with a specific goal to be faster.  
  
  ---
  
  ### Answer by username_1 on 2017-05-19. Score: 8
  
  Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency. So `samtools merge` (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), `picard MergeSamFiles`, etc.
  
  These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors. Use `ulimit` to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc. Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.
  
  > **Comments:**
  > **username_2**: Maybe merging stepwise as you mention in your second paragraph could speed up things as well as one doesn't hit IO limitations?  
  > **username_1**: No, some of the data is now being written and read more than once, so it must be slower.  With crafty use of pipes and process substitution (`samtools merge … <(samtools merge …) …`) this can be minimised and avoid hitting disk, so the slowdown can be trivial (at the cost of using a few more cores).  
  
  ---
  
  