Answers.Body: <p>It's unlikely that two different mapping tools will give exactly the same alignment, scores, and match strings for the same sequence mapped to the same reference. For some sequence/reference alignments, it's impossible to determine which is the "best" alignment, and slight differences in code can have large effects on the chosen alignment.</p>
  
  <p>However, even if the actual mapping location and match string are exactly the same (for example when using a tool like Picard to filter BAM/SAM files), different tools will incorporate different metadata with each mapping. This is allowed in the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="nofollow noreferrer">SAM file format specification</a> by the addition of optional fields beyond the 11th column. There are a few standard optional tags that can be used in these fields, and additional custom non-standard tags can be used as well. It is very likely that Picard is adding additional metadata to the alignments in the BAM/SAM file.</p>
  
  <p>There's an additional complication in that the underlying <em>SAM</em> alignment (and metadata) could be identical, but the <em>BAM</em> file can still have different file sizes. One reason for this is that BAM file compression methods can be changed. For example, alignment tools might choose a quick compression method, while filtering tools might choose a method that results in better compression.</p>
  
  <p>Checking for alignment similarity is more difficult than just comparing files at a binary level, and your particular application (or context, or story) will change what the best method of comparison is. It would be helpful to know why you want to compare BAM files in order to provide a better answer to your question.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-03T00:20:02.360Z
Answers.LastActivityDate: 2017-06-03T00:20:02.360Z
Answers.UserDisplayName: gringer
Answers.Body: <p>It's worth bearing in mind that when outputting compressed BAM, as most tools do by default, they may well be using different levels of compression and/or different libraries, or versions of said, libraries for doing (de)compression which will result in different file sizes.  Additionally coordinate sorted BAM will compress more than unsorted BAM. The current version of Picard uses <a href="https://github.com/samtools/htsjdk" rel="noreferrer">HTSJDK</a> which in turn uses java.util.zip.Deflater/Inflater, current versions of samtools should be using the <a href="https://github.com/samtools/htslib" rel="noreferrer">HTSlib</a> library which in turn depends on the <a href="http://zlib.net/" rel="noreferrer">standard zlib library</a>. You can see the effect of different implementations of zlib have on file size and execution time in benchmarking done by the <a href="http://www.htslib.org/benchmarks/zlib.html" rel="noreferrer">samtools team</a>.</p>
  
  <p>However, in your case the best way to see if there is any difference between the BAM files is to rule out the effect of different levels of compression or libraries used for compression and save both BAM files as uncompressed.  Both samtools and Picard have options for disabling or changing the levels of compression, since the BAM compression standard BGZF is implemented on top of the gzip format it has inherited the ability, just like with gzip, to change the compression level from 0 to 9.</p>
  
  <p><code>samtools view -bu</code> will allow you to produce uncompressed BAM output (which is also handy for piping into other programs as it saves time wasted compressing decompressing what is essentially a stream).  Also note that <code>samtools sort</code> has a <code>-l INT</code> setting where INT can be set between 0 (compression off, as with <code>-u</code>) 1 (for fastest compression, but increased file size) or -9 (for maximal compression, with increased run time).  Some of the effects of increased runtime for higher compression settings might be ameliorated using the <code>-@</code> argument which allows you to set the number of extra threads used for BAM compression, by default samtools won't use any.  </p>
  
  <p>Picard tools has a general setting <code>COMPRESSION_LEVEL</code> which is applicable to most of its tools setting this to 0, <code>COMPRESSION_LEVEL=0</code> should disable compression. </p>
  
  <p>So re-running whatever Picard tool you used in the first instance with <code>COMPRESSION_LEVEL=0</code> will then enable you to check that the file is not further modified by <code>samtools view -bu</code>. The assumption here being that if both files have exactly the same content they should be the same size uncompressed, of course if they have trivial differences with regard to white space formatting things may still differ.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-03T12:14:11.773Z
Answers.LastActivityDate: 2017-06-03T12:14:11.773Z
Answers.UserDisplayName: Matt Bashton
Body: <p>I have a BAM created by Picard. I want to filter alignments by flags with <code>samtools view</code>. However, I noticed that even if I apply no filters, the output BAM is different from my input BAM. Are BAMs produced by different tools also different in size? How can I check if two BAMs are the same?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-02T23:50:55.187Z
Comments.Text: Hi medbe, thanks for asking your question, which fits well with this bioinformatics site. To improve the answers that you will get, it can be helpful to edit your question and build a bit of a story around it. What led you to want to filter alignments by flags? Why does it matter that the BAM files have a different format?
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-03T11:38:04.707Z
Comments.Text: Could you elaborate on exactly how samtools/picard output is different? Is it just the file size or are you missing data from the optional fields of the alignment section?
Comments.UserDisplayName: Matt Bashton
Tags: <sam>
Title: Why do BAM files created by different tools have different file sizes?
ThreadText: # Why do BAM files created by different tools have different file sizes?
  
  (Asked by: username_0 on 2017-06-02)
  
  I have a BAM created by Picard. I want to filter alignments by flags with `samtools view`. However, I noticed that even if I apply no filters, the output BAM is different from my input BAM. Are BAMs produced by different tools also different in size? How can I check if two BAMs are the same?
  
  **Tags:** sam
  
  > **Comments:**
  > **username_1**: Hi username_0, thanks for asking your question, which fits well with this bioinformatics site. To improve the answers that you will get, it can be helpful to edit your question and build a bit of a story around it. What led you to want to filter alignments by flags? Why does it matter that the BAM files have a different format?  
  > **username_2**: Could you elaborate on exactly how samtools/picard output is different? Is it just the file size or are you missing data from the optional fields of the alignment section?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-06-03. Score: 7
  
  It's worth bearing in mind that when outputting compressed BAM, as most tools do by default, they may well be using different levels of compression and/or different libraries, or versions of said, libraries for doing (de)compression which will result in different file sizes. Additionally coordinate sorted BAM will compress more than unsorted BAM. The current version of Picard uses [HTSJDK](https://github.com/samtools/htsjdk) which in turn uses java.util.zip.Deflater/Inflater, current versions of samtools should be using the [HTSlib](https://github.com/samtools/htslib) library which in turn depends on the [standard zlib library](http://zlib.net/). You can see the effect of different implementations of zlib have on file size and execution time in benchmarking done by the [samtools team](http://www.htslib.org/benchmarks/zlib.html).
  
  However, in your case the best way to see if there is any difference between the BAM files is to rule out the effect of different levels of compression or libraries used for compression and save both BAM files as uncompressed. Both samtools and Picard have options for disabling or changing the levels of compression, since the BAM compression standard BGZF is implemented on top of the gzip format it has inherited the ability, just like with gzip, to change the compression level from 0 to 9.
  
  `samtools view -bu` will allow you to produce uncompressed BAM output (which is also handy for piping into other programs as it saves time wasted compressing decompressing what is essentially a stream). Also note that `samtools sort` has a `-l INT` setting where INT can be set between 0 (compression off, as with `-u`) 1 (for fastest compression, but increased file size) or -9 (for maximal compression, with increased run time). Some of the effects of increased runtime for higher compression settings might be ameliorated using the `-@` argument which allows you to set the number of extra threads used for BAM compression, by default samtools won't use any.
  
  Picard tools has a general setting `COMPRESSION_LEVEL` which is applicable to most of its tools setting this to 0, `COMPRESSION_LEVEL=0` should disable compression.
  
  So re-running whatever Picard tool you used in the first instance with `COMPRESSION_LEVEL=0` will then enable you to check that the file is not further modified by `samtools view -bu`. The assumption here being that if both files have exactly the same content they should be the same size uncompressed, of course if they have trivial differences with regard to white space formatting things may still differ.
  
  
  ---
  
  ### Answer by username_1 on 2017-06-03. Score: 4
  
  It's unlikely that two different mapping tools will give exactly the same alignment, scores, and match strings for the same sequence mapped to the same reference. For some sequence/reference alignments, it's impossible to determine which is the "best" alignment, and slight differences in code can have large effects on the chosen alignment.
  
  However, even if the actual mapping location and match string are exactly the same (for example when using a tool like Picard to filter BAM/SAM files), different tools will incorporate different metadata with each mapping. This is allowed in the [SAM file format specification](https://samtools.github.io/hts-specs/SAMv1.pdf) by the addition of optional fields beyond the 11th column. There are a few standard optional tags that can be used in these fields, and additional custom non-standard tags can be used as well. It is very likely that Picard is adding additional metadata to the alignments in the BAM/SAM file.
  
  There's an additional complication in that the underlying *SAM* alignment (and metadata) could be identical, but the *BAM* file can still have different file sizes. One reason for this is that BAM file compression methods can be changed. For example, alignment tools might choose a quick compression method, while filtering tools might choose a method that results in better compression.
  
  Checking for alignment similarity is more difficult than just comparing files at a binary level, and your particular application (or context, or story) will change what the best method of comparison is. It would be helpful to know why you want to compare BAM files in order to provide a better answer to your question.
  
  
  ---
  
  