Answers.Body: <p>The main concern has always been the "binning" of quality scores that occurs via CRAM compression (and is also standard on the HiSeqX, HiSeq4000, and NovaSeq platforms). Anecdotally, I can report very little difference between 4-bin quality scores and full quality scores on cancer samples, though I don't know if I've seen a direct head-to-head comparison. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-09T16:32:37.037Z
Answers.Text: +1. My colleagues have done some benchmarks to show 4-bin has little effect on *germline* samples. I have seen the similar. Cancer samples always make me wary, though. It would be great if someone do a systematic evaluation on cancer samples. I haven't seen one so far.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-09T16:28:26.360Z
Answers.LastActivityDate: 2017-06-09T16:28:26.360Z
Answers.UserDisplayName: chrisamiller
Answers.Body: <p>By default, a CRAM you create with samtools is lossless. It typically halves the input BAM in terms of file size. If you want to compress more, you can let samtools convert most read names to integers. You won't be able to tell optical duplicates from read names, but this is a minor concern. You can also drop useless tags depending on your mapper and the downstream caller in use. For cancer data, I wouldn't reduce the resolution of base quality without comprehensive benchmarks. Unfortunately, base quality takes most of space in CRAM. Discarding the original read names and some tags probably won't save you much space.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:24:48.840Z
Answers.Text: These are all great suggestions for reducing file size without losing information, but don't address the main question: the effect of lossiness on variant calls.
Answers.UserDisplayName: Daniel Standage
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:27:14.510Z
Answers.Text: @DanielS If you don't touch bases and qualities and name pairing, you won't change the variant calls.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:30:41.023Z
Answers.Text: Yes, but then that's not really lossy, is it? Doesn't lossy compression conventionally involve changing the sequence and/or quality values for greater compression efficiency?
Answers.UserDisplayName: Daniel Standage
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:41:27.150Z
Answers.Text: That depends on the definition of "lossy" :) To me, losing read names and tags is lossy.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:49:53.817Z
Answers.Text: ¯\\_(ツ)_/¯ You yourself said that calls shouldn't change if the sequence and quality are unchanged. So everything else is ancillary. Don't get me wrong, I think it's valuable to point out that it's possible to reduce file size without changing sequence or quality, but it seemed pretty clear to me that the OP was talking about lossy compression of the sequence and/or quality scores.
Answers.UserDisplayName: Daniel Standage
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:51:38.977Z
Answers.Text: Then again, this answer is a reasonable response to the question "Which compression strategy has the lowest impact?" Ok, I take it all back! :-)
Answers.UserDisplayName: Daniel Standage
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-08T16:20:00.480Z
Answers.LastActivityDate: 2017-06-08T16:20:00.480Z
Answers.UserDisplayName: user172818
Body: <p>We're considering switching our storage format from BAM to CRAM. We work with human cancer samples, which may have very low prevalence variants (i.e. not diploid frequency).</p>
  
  <p>If we use lossy CRAM to save more space, how much will variants called from those CRAM files change? Which compression strategy has the lowest impact? </p>
  
  <p>Are there any other impacts on downstream tools that we're not considering?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T14:58:54.587Z
Comments.Text: CRAM doesn't **need** to be lossy, is there a reason you need it to be?
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T15:02:49.157Z
Comments.Text: Saving disk space. We pay by the GB and need to keep the data around for 10 years.
Comments.UserDisplayName: morgantaschuk
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T15:08:59.283Z
Comments.Text: Can't argue that budget isn't a good reason :)
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T15:16:31.463Z
Comments.Text: Interesting question. I think this is the kind of thing that makes a nice side project. Take a bam file call the variants, transform it to cram and run the variant caller. Measure the difference and the variant concordance between the two approaches using a number of different files.
Comments.UserDisplayName: eastafri
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T16:54:31.743Z
Comments.Text: If one is concerned about absolute integrity/reproducibility of the data then budget consideration is not a good reason.
Comments.UserDisplayName: GenoMax
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T22:08:54.607Z
Comments.Text: Unless the answer is: some types of lossy compression don't have any impact on variant calls.
Comments.UserDisplayName: morgantaschuk
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-08T22:32:15.993Z
Comments.Text: BAM files are zipped with standard gzip compression. Unzip them to "naked BAM" - not my terminology - and re-zip them with something stronger like 7zip/LZMA. You can always re-zip them again with the bgzip tool when you need them back in true BAM format again. This gets you most of the way there filesize-wise without really changing the format, which could be good if you've got things set up how you like. Not an answer because it doesn't answer your question but it might solve your problem.
Comments.UserDisplayName: J.J
Tags: <sam><file-formats><variant-calling><storage>
Title: Do variant calls change when you call from CRAM?
ThreadText: # Do variant calls change when you call from CRAM?
  
  (Asked by: username_0 on 2017-06-08)
  
  We're considering switching our storage format from BAM to CRAM. We work with human cancer samples, which may have very low prevalence variants (i.e. not diploid frequency).
  
  If we use lossy CRAM to save more space, how much will variants called from those CRAM files change? Which compression strategy has the lowest impact?
  
  Are there any other impacts on downstream tools that we're not considering?
  
  **Tags:** sam, file-formats, variant-calling, storage
  
  > **Comments:**
  > **username_1**: CRAM doesn't **need** to be lossy, is there a reason you need it to be?  
  > **username_0**: Saving disk space. We pay by the GB and need to keep the data around for 10 years.  
  > **username_1**: Can't argue that budget isn't a good reason :)  
  > **username_2**: Interesting question. I think this is the kind of thing that makes a nice side project. Take a bam file call the variants, transform it to cram and run the variant caller. Measure the difference and the variant concordance between the two approaches using a number of different files.  
  > **username_3**: If one is concerned about absolute integrity/reproducibility of the data then budget consideration is not a good reason.  
  > **username_0**: Unless the answer is: some types of lossy compression don't have any impact on variant calls.  
  > **username_4**: BAM files are zipped with standard gzip compression. Unzip them to "naked BAM" - not my terminology - and re-zip them with something stronger like 7zip/LZMA. You can always re-zip them again with the bgzip tool when you need them back in true BAM format again. This gets you most of the way there filesize-wise without really changing the format, which could be good if you've got things set up how you like. Not an answer because it doesn't answer your question but it might solve your problem.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_6 on 2017-06-08. Score: 6 | Accepted answer
  
  By default, a CRAM you create with samtools is lossless. It typically halves the input BAM in terms of file size. If you want to compress more, you can let samtools convert most read names to integers. You won't be able to tell optical duplicates from read names, but this is a minor concern. You can also drop useless tags depending on your mapper and the downstream caller in use. For cancer data, I wouldn't reduce the resolution of base quality without comprehensive benchmarks. Unfortunately, base quality takes most of space in CRAM. Discarding the original read names and some tags probably won't save you much space.
  
  > **Comments:**
  > **username_7**: These are all great suggestions for reducing file size without losing information, but don't address the main question: the effect of lossiness on variant calls.  
  > **username_6**: @DanielS If you don't touch bases and qualities and name pairing, you won't change the variant calls.  
  > **username_7**: Yes, but then that's not really lossy, is it? Doesn't lossy compression conventionally involve changing the sequence and/or quality values for greater compression efficiency?  
  > **username_6**: That depends on the definition of "lossy" :) To me, losing read names and tags is lossy.  
  > **username_7**: ¯\\_(ツ)_/¯ You yourself said that calls shouldn't change if the sequence and quality are unchanged. So everything else is ancillary. Don't get me wrong, I think it's valuable to point out that it's possible to reduce file size without changing sequence or quality, but it seemed pretty clear to me that the OP was talking about lossy compression of the sequence and/or quality scores.  
  > **username_7**: Then again, this answer is a reasonable response to the question "Which compression strategy has the lowest impact?" Ok, I take it all back! :-)  
  
  ---
  
  ### Answer by username_5 on 2017-06-09. Score: 3
  
  The main concern has always been the "binning" of quality scores that occurs via CRAM compression (and is also standard on the HiSeqX, HiSeq4000, and NovaSeq platforms). Anecdotally, I can report very little difference between 4-bin quality scores and full quality scores on cancer samples, though I don't know if I've seen a direct head-to-head comparison.
  
  > **Comments:**
  > **username_6**: +1. My colleagues have done some benchmarks to show 4-bin has little effect on *germline* samples. I have seen the similar. Cancer samples always make me wary, though. It would be great if someone do a systematic evaluation on cancer samples. I haven't seen one so far.  
  
  ---
  
  