Answers.Body: <p>EDIT: I am rewriting the answer in response to updates to the original question.</p>
  <p><strong>TL;DR</strong>: use CRAM</p>
  <h3>Background 1: quality binning and FASTQ compression</h3>
  <p>In the old days, base callers outputted base quality at full resolution – you could see quality from Q2 to Q40 in full range. As a result, quality strings were like semi-random strings and very difficult to compress. Later people gradually realized that keeping base quality in low resolution wouldn't affect downstream analysis. The Illumina basecaller started to output quality in 8 distinct values and later changed that to 4 bins. This change greatly simplified quality string and made them compressed better. For example, in old days, a 30X BAM would take ~100 GB. With quality binning, it would only take ~60 GB.</p>
  <h3>Background 2: GATK Base quality recalibration</h3>
  <p>In early 2010s, Illumina base quality was not calibrated well. GATK people introduced BQSR to correct that and observed noticeable improvement in SNP accuracy. Nonetheless, with improved Illumina base caller, their base quality became more accurate. Meanwhile, the world moved to 30X deep sequencing. The depth overwhelms slight inaccuracy in quality. I would say around 2015, BQSR was already unnecessary for data produced at the time.</p>
  <p>Does it hurt to apply BQSR? Yes. First, BQSR introduces subtle biases towards the reference and towards known SNPs. Second, BQSR distorts the data. At least for some datasets, I observed that SNP accuracy dropped with variant quality after BQSR; I didn't observe this with raw quality. Third, BQSR is slow. Fourth, for new sequencers producing data at higher quality, BQSR is likely to decrease data quality. Last, related to the question, BQSR added another semi-random quality string and made compression even harder. Nowadays, running BQSR is a waste of resource for worse results. The official GATK best practice no longer uses BQSR according to their <a href="https://github.com/gatk-workflows/gatk4-germline-snps-indels/blob/master/haplotypecaller-gvcf-gatk4.wdl" rel="noreferrer">WDL file</a>.</p>
  <h3>CRAM</h3>
  <p>These days a 30X human CRAM only takes ~15 GB (see <a href="https://ftp.sra.ebi.ac.uk/vol1/run/ERR324/ERR3240144/" rel="noreferrer">this file</a>). This is a huge contrast to ~100 GB BAM in early 2010s. OP only saw ~10% saving probably due to a) BQSR and/or b) old data with full quality resolution.</p>
  <p>On encoding/decoding speed, CRAM was much slower than BAM. Not any more. The latest htslib implementation of CRAM is faster than BAM on encoding and only slightly slower on decoding. The poor performance of IGV on CRAM could be that the Java CRAM decoder is not as optimized.</p>
  <p>It is true that CRAM is not as widely supported as BAM. However, all the other alternatives are much worse. Petagene said they had <a href="https://www.petagene.com/wp-content/uploads/2019/04/PetaGene-IGV-Windows-Mac-Press-Release-FINAL.pdf" rel="noreferrer">IGV-PG</a> (PDF) for their format. That is not official IGV and I couldn't find more recent update beyond the 2019 press release. I don't see other viable options.</p>
  <p>Note that the common practice is to keep all raw reads, mapped or not, in BAM or CRAM such that you can get raw reads back later. BAM/CRAM additionally keeps metadata like read group, sample name, run information etc and is actually more popular than FASTQ in large sequencing centers. Also note that you don't need to sort CRAM by coordinate. Unsorted CRAM is only a little larger than sorted CRAM.</p>
  <h3>CRAM and its competitors</h3>
  <p>The core CRAM developer, James Bonfield, is one of the most knowledgeable researchers on compression (and one of the best C programmers) in this field. He has done a lot of compression evaluation over the years. The conclusion is that on a fair benchmark, CRAM is comparable to the best tools so far in terms of compression ratio.</p>
  <p>Petagene could compress better in the plot @terdon showed mostly because it has a special treatment of the <code>OQ</code> tag generated by GATK BQSR. It is a typical trick marketing people use to make their methods look better. With BQSR phased out, this plot is no longer relevant.</p>
  <h3>On commercial software</h3>
  <p>In general, I welcome commercial tools and think they are invaluable to users. I also have huge respect to DRAGEN developers. However, on FASTQ storage, I would strongly recommend against closed-source compressors. If those tools go under, you may lose your data. Not worth it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T09:55:25.667Z
Answers.Text: Thanks for your input, but mainly for the link. WDL seemed too complicated for me. We also noticed that BQSR was not recommended by nf-core people as well, but none provided a concrete reason beyond "its not useful". As I've also commented on @gringer post, CRAM didn't work well when using it with IGV, as well as some downstream tools don't work with CRAM.
Answers.UserDisplayName: Karthik Nair
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T15:07:31.707Z
Answers.Text: @KarthikNair It is interesting that you are criticizing CRAM for slowness in IGV, while you are compressing fastq with spring and you can't even load other alignment formats into IGV. What are you expecting? Also, you don't need to understand WDL. Just search/grep "BaseRecalibrator" or "BQSR" to see if it gets called.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T15:34:37.867Z
Answers.Text: @KarthikNair sadly, the [GATK best practices](https://tinyurl.com/ykn3bncp), which were updated very recently, still recommend BQSR so it is really surprising that their recommended pipeline doesn't use it. That said, in my place of work, we analyze around 2k human samples a month, and we have found that BQSR often causes us to miss variants. I don't have statistics for this yet so it's anecdotal, but there is strong evidence that BQSR not only isn't particularly helpful today, but can be actively harmful. And not only with small panels, this also happens with whole exome sequencing samples.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T15:49:58.337Z
Answers.Text: @terdon That page is referencing their older pipelines, which do use BQSR. The WDL file in my link is two years old. It is the latest I can find. IMO, BQSR has been hurting for 10 years. Well, better late than never.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T17:02:24.860Z
Answers.Text: you focus on BAM but what would you suggest for FASTQ if not DRAGEN ORA reference based approach? Arguably long term of storage of FASTQ is more important since BAMs can be regenerated
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T17:41:12.047Z
Answers.Text: @Chris_Rands See new edits. I see the value in reference-free algorithms such as spring and dsrc. But if you already require a reference, it is better to keep CRAM than ORA.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T18:21:22.773Z
Answers.Text: For what it's worth, petagene's compatibility layer lets both IGV and JBrowse work out of the box with peta-compressed bam files. You don't even need to use their IGV implementation, vanilla IGV works fine as long as you are using petagene's trick with the symlinks I mentioned in my answer.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T21:30:07.813Z
Answers.Text: @Chris_Rands You can align raw reads with STAR or any local aligner and then create a CRAM. I guess that will give you a smaller file than BAM.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-13T04:55:51.717Z
Answers.Text: Hay all, thanks for all your comments! I will echo @terdon and encourage people to share their findings w.r.t. to BQSR not being very helpful. We have been using MarkDuplicatesSpark and BQSR pipeline. I'm planning on dropping the BQSR steps and using samtools markdup -s instead of MarkDuplicatesSpark (Mainly for optimization). And yes, perhaps the lackluster size reduction could have been due to BQSR in data. We also have old NGS data sequenced from 2016 onwards, so some samples might use unbinned quality scores.
Answers.UserDisplayName: Karthik Nair
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T23:32:23.050Z
Answers.LastActivityDate: 2023-04-12T17:38:32.693Z
Answers.LastEditDate: 2023-04-12T17:38:32.693Z
Answers.UserDisplayName: user172818
Answers.Body: <p>I'd recommend reconsidering CRAM. It's an open standard that is most likely to still be accessible many years in the future. Regarding your point about lossy compression...</p>
  <blockquote>
  <p>Can't use lossy [CRAM] compression as these BAM files might be subject to future downstream analysis (Ex. CNV calling, variant calling or something else).</p>
  </blockquote>
  <p>By default, CRAM compression is not lossy.</p>
  <p>CRAM does, however, have an optional archive settings mode (<code>samtools view --output-fmt cram --output-fmt-option archive ...</code>) which <em>is</em> a lossy compression, doing things like removing read names, removing additional accessory fields, and additional compression of quality scores. In all cases, the base sequence of the reads is preserved.</p>
  <p><a href="https://www.htslib.org/doc/samtools.html" rel="noreferrer">https://www.htslib.org/doc/samtools.html</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T20:49:16.553Z
Answers.LastActivityDate: 2023-04-11T20:49:16.553Z
Answers.UserDisplayName: gringer
Answers.Body: <p>An alternative to the good option of PetaGene that @terdon mentioned and also not open source is the <a href="https://developer.illumina.com/news-updates/introducing-dragen-original-read-archive-ora" rel="nofollow noreferrer">DRAGEN ORA compression</a>. This is loseless reference based compression and can be ~5 x smaller over fastq.gz, but it's only for fastq files and only those generated by Illumina machines (and will be easiest within the DRAGEN ecosystem).</p>
  <p>Another point to consider is where you store the data. Even with 'standard' compression you might get more benefits moving archival data to a system like <a href="https://en.wikipedia.org/wiki/Amazon_S3_Glacier" rel="nofollow noreferrer">AWS S3 Glacier</a>, depending on your current infastructure setup of course</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T15:18:54.923Z
Answers.Text: This is exactly what I do. I archive FASTQ.GZ (with added content-md5 checksum) to S3. I also archive processed outputs to a different location that is less fault tolerant than S3 as those files can be recreated.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T09:41:11.743Z
Answers.Text: We did check this out, but we do not sequence the NGS data ourselves. We ask external labs to perform the sequencing and they share the fastq.gz files with us. What is the approximate pricing for this? I can't clearly identify where those details are!
Answers.UserDisplayName: Karthik Nair
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T14:11:18.687Z
Answers.LastActivityDate: 2023-04-11T14:11:18.687Z
Answers.UserDisplayName: Chris_Rands
Answers.Body: <p>The only free and open source tool I know that can help is <a href="https://en.wikipedia.org/wiki/Zstd" rel="nofollow noreferrer"><code>zstd</code></a>. Their github repository's README describes it as:</p>
  <blockquote>
  <p>Zstandard, or zstd as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.</p>
  </blockquote>
  <p>We read <a href="http://www.bioinformaticszen.com/post/use-zstd-for-raw-fastq/" rel="nofollow noreferrer">this blog post</a> on using it to compress fastq files and were intrigued so we ran some tests. We couldn't reproduce the same level of compression that the article claims but that might be due to different input files or the way an index was/wasn't created.</p>
  <p>Even in our tests though, we did see that it provides better compression than gzip. Here are the specific values for one of the tests, run using an R1 file of a human WGS run:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: center;">Uncompressed fastq</th>
  <th style="text-align: left;">bgzip-compressed</th>
  <th style="text-align: center;">zstd-compressed</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: center;">179G</td>
  <td style="text-align: left;">58G</td>
  <td style="text-align: center;">47G</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>This was done using <code>-z 19</code>, to set the compression level to 19 (we found this to be the best tradeoff between size and time to compress). We also saw that using a human genome fasta file to create a compression index, and then using that when compressing human fastq files, actually resulted in worse performance (larger compressed file) than not using the index. But it is entirely possible that we just did it wrong, we didn't spend a lot of time on it.</p>
  <p>So you might want to investigate <code>zstd</code> as well, maybe you can get better performance if you play around with the indexing. On the plus side, it is free
  and open source. On the other hand, it won't do much with bam or cram files.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T09:37:29.567Z
Answers.Text: Thanks for the input. I'm pretty sure that with Spring, the output size might be around 20-30 GB, if not slightly lesser. Granted, you would have to compress / decompress the data, but the final file size would be pretty small. We have compressed WGS samples with spring, it goes from total 50GB of gz files to around 15-20 GB in final spring file.
Answers.UserDisplayName: Karthik Nair
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-18T12:02:34.410Z
Answers.Text: Zstd (and Zpaq and any other general compression tool) deliver relatively bad performance on SAM/BAM files. This is because they are either LZ-based or otherwise view the text as being "one long text", whereas SAM/BAM is anything but that. 
  
  FYI, this opinion is based on fairly thorough testing.
Answers.UserDisplayName: Marco
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T18:10:10.360Z
Answers.LastActivityDate: 2023-04-14T11:43:13.767Z
Answers.LastEditDate: 2023-04-14T11:43:13.767Z
Answers.UserDisplayName: terdon
Answers.Body: <p>You might want to consider <a href="https://www.petagene.com/" rel="nofollow noreferrer">PetaGene</a>. This is a paid tool, and not open source, but it is something we use at work and it is really good! It can do lossless compression of gzip-compressed fastq files, bam files, and even cram files, and achieves really good size reduction (image from the PetaGene webpage):</p>
  <p><a href="https://i.sstatic.net/8JqUZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8JqUZ.png" alt="petagene compression performance bar graph" /></a></p>
  <p>In my own testing, using files of different sizes, I get an average size reduction of 62.1% (2.6 x smaller) for fastq.gz files (compared to the gzipped fastq file, not compared to the original, uncompressed fastq file) and an average reduction of 82.1% (5.6 x smaller) for bam files. I didn't test cram files.</p>
  <p>If you have the budget for a paid service, these guys do good work! The greatest advantage, apart from the very significant size reduction, is that they also provide a system that allows you to use the compressed files natively. So you can feed the petagene-compressed bam file directly to bwa for example. They create special symlinks (<code>foo.fastq.gz -&gt; foo.fastq.petacompressed</code>) and you can then use these symlinks and petagene handles the decompression on the fly.</p>
  <hr />
  <p>The <a href="https://en.wikipedia.org/wiki/CRAM_(file_format)" rel="nofollow noreferrer">Wikipedia page on CRAM</a> has a nice table with a few alternatives to CRAM which I reproduce below. I haven't tried any except PetaGene though so cannot comment on their performance:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Name</th>
  <th>Key advantage over CRAM</th>
  <th>Repo</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Deez</td>
  <td>None. An early approach to SAM compression, introduced 2014.</td>
  <td><a href="https://github.com/sfu-compbio/deez" rel="nofollow noreferrer">https://github.com/sfu-compbio/deez</a></td>
  </tr>
  <tr>
  <td>Genozip</td>
  <td>Significantly better compression, but slower random access</td>
  <td><a href="https://github.com/divonlan/genozip" rel="nofollow noreferrer">https://github.com/divonlan/genozip</a></td>
  </tr>
  <tr>
  <td>GenomSys</td>
  <td>Utilizes ISO-standard MPEG-G, but compression inferior to CRAM 3.1</td>
  <td>Source code not publicly available</td>
  </tr>
  <tr>
  <td>PetaSuite</td>
  <td>Virtual filesystem interface</td>
  <td>Source code not publicly available</td>
  </tr>
  </tbody>
  </table>
  </div>
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T09:43:02.890Z
Answers.Text: We considered this software, but sadly we don't have the budget for this. And yes, on-the-fly decompression is a great feature to have, makes storing all data in this format extremely viable. Right now, we get decent compression with spring, but it needs lots of de/compression time, not to mention the CPU resources it guzzles during the process lol
Answers.UserDisplayName: Karthik Nair
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T15:12:04.423Z
Answers.Text: @terdon genozip is quality work and is very impressive. Its full source code is also on github. If genozip goes under some day (I hope not!), you may create a fork and continue to use their format.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-29T08:17:16.340Z
Answers.Text: @user172818 ~terdon - can you share any further info on how PetaGene achieves compression and the critique mentioned of "special treatment of the OQ tag generated by GATK BQSR"? I know it's closed source but trying to understand conceptually how their compression works
Answers.UserDisplayName: Chris_Rands
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-29T08:22:43.257Z
Answers.Text: Not I, @Chris_Rands, sorry. I'm sure user172818 will know far more, but I have no idea and wasn't even aware there was any special treatment before reading their answer. I know at least one person who used to be an engineer working for them is active on the site. If there's something that can be shared, maybe they will answer if you post a question.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T12:09:29.343Z
Answers.LastActivityDate: 2023-05-23T11:26:05.740Z
Answers.LastEditDate: 2023-05-23T11:26:05.740Z
Answers.UserDisplayName: terdon
Body: <p>We have a lot of Illumina sequenced exome data. Currently we are using <a href="https://github.com/shubhamchandak94/Spring" rel="noreferrer">spring</a> for its great lossless compression, but we are looking if there is anything better (and most preferably opensource) which can let us compress our fastq files.</p>
  <p>We also want to compress or reduce file size of BAM, but using CRAM in lossless mode didn't yield good results. The space savings were decent, roughly ~10% (IMO was expecting more), but the biggest hit came when using it with IGV. We have set up a http-server and stream bams to IGV (Helped us immensely as people wouldn't have to download and visualize bam files). With Cram, the loading times were significantly high, requiring users to commit more memory for IGV. Sometimes we would have to wait around a minute for reads to show up. With Bam it is near instantaneous.</p>
  <p>Can't use lossy compression as these BAM files might be subject to future downstream analysis (Ex. CNV calling, variant calling or something else).</p>
  <p>What are some good ways to compress fastq and bam files so we can maximize our storage?</p>
  
Tags: <sam><fastq><cram><storage>
Title: Good / recommended way to archive fastq and bam files?
ThreadText: # Good / recommended way to archive fastq and bam files?
  
  (Asked by: username_0 on 2023-04-11)
  
  We have a lot of Illumina sequenced exome data. Currently we are using [spring](https://github.com/shubhamchandak94/Spring) for its great lossless compression, but we are looking if there is anything better (and most preferably opensource) which can let us compress our fastq files.
  
  We also want to compress or reduce file size of BAM, but using CRAM in lossless mode didn't yield good results. The space savings were decent, roughly ~10% (IMO was expecting more), but the biggest hit came when using it with IGV. We have set up a http-server and stream bams to IGV (Helped us immensely as people wouldn't have to download and visualize bam files). With Cram, the loading times were significantly high, requiring users to commit more memory for IGV. Sometimes we would have to wait around a minute for reads to show up. With Bam it is near instantaneous.
  
  Can't use lossy compression as these BAM files might be subject to future downstream analysis (Ex. CNV calling, variant calling or something else).
  
  What are some good ways to compress fastq and bam files so we can maximize our storage?
  
  **Tags:** sam, fastq, cram, storage
  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_1 on 2023-04-11. Score: 14
  
  EDIT: I am rewriting the answer in response to updates to the original question.
  
  **TL;DR**: use CRAM
  
  ### Background 1: quality binning and FASTQ compression
  
  In the old days, base callers outputted base quality at full resolution – you could see quality from Q2 to Q40 in full range. As a result, quality strings were like semi-random strings and very difficult to compress. Later people gradually realized that keeping base quality in low resolution wouldn't affect downstream analysis. The Illumina basecaller started to output quality in 8 distinct values and later changed that to 4 bins. This change greatly simplified quality string and made them compressed better. For example, in old days, a 30X BAM would take ~100 GB. With quality binning, it would only take ~60 GB.
  
  ### Background 2: GATK Base quality recalibration
  
  In early 2010s, Illumina base quality was not calibrated well. GATK people introduced BQSR to correct that and observed noticeable improvement in SNP accuracy. Nonetheless, with improved Illumina base caller, their base quality became more accurate. Meanwhile, the world moved to 30X deep sequencing. The depth overwhelms slight inaccuracy in quality. I would say around 2015, BQSR was already unnecessary for data produced at the time.
  
  Does it hurt to apply BQSR? Yes. First, BQSR introduces subtle biases towards the reference and towards known SNPs. Second, BQSR distorts the data. At least for some datasets, I observed that SNP accuracy dropped with variant quality after BQSR; I didn't observe this with raw quality. Third, BQSR is slow. Fourth, for new sequencers producing data at higher quality, BQSR is likely to decrease data quality. Last, related to the question, BQSR added another semi-random quality string and made compression even harder. Nowadays, running BQSR is a waste of resource for worse results. The official GATK best practice no longer uses BQSR according to their [WDL file](https://github.com/gatk-workflows/gatk4-germline-snps-indels/blob/master/haplotypecaller-gvcf-gatk4.wdl).
  
  ### CRAM
  
  These days a 30X human CRAM only takes ~15 GB (see [this file](https://ftp.sra.ebi.ac.uk/vol1/run/ERR324/ERR3240144/)). This is a huge contrast to ~100 GB BAM in early 2010s. OP only saw ~10% saving probably due to a) BQSR and/or b) old data with full quality resolution.
  
  On encoding/decoding speed, CRAM was much slower than BAM. Not any more. The latest htslib implementation of CRAM is faster than BAM on encoding and only slightly slower on decoding. The poor performance of IGV on CRAM could be that the Java CRAM decoder is not as optimized.
  
  It is true that CRAM is not as widely supported as BAM. However, all the other alternatives are much worse. Petagene said they had [IGV-PG](https://www.petagene.com/wp-content/uploads/2019/04/PetaGene-IGV-Windows-Mac-Press-Release-FINAL.pdf) (PDF) for their format. That is not official IGV and I couldn't find more recent update beyond the 2019 press release. I don't see other viable options.
  
  Note that the common practice is to keep all raw reads, mapped or not, in BAM or CRAM such that you can get raw reads back later. BAM/CRAM additionally keeps metadata like read group, sample name, run information etc and is actually more popular than FASTQ in large sequencing centers. Also note that you don't need to sort CRAM by coordinate. Unsorted CRAM is only a little larger than sorted CRAM.
  
  ### CRAM and its competitors
  
  The core CRAM developer, James Bonfield, is one of the most knowledgeable researchers on compression (and one of the best C programmers) in this field. He has done a lot of compression evaluation over the years. The conclusion is that on a fair benchmark, CRAM is comparable to the best tools so far in terms of compression ratio.
  
  Petagene could compress better in the plot @username_2 showed mostly because it has a special treatment of the `OQ` tag generated by GATK BQSR. It is a typical trick marketing people use to make their methods look better. With BQSR phased out, this plot is no longer relevant.
  
  ### On commercial software
  
  In general, I welcome commercial tools and think they are invaluable to users. I also have huge respect to DRAGEN developers. However, on FASTQ storage, I would strongly recommend against closed-source compressors. If those tools go under, you may lose your data. Not worth it.
  
  > **Comments:**
  > **username_0**: Thanks for your input, but mainly for the link. WDL seemed too complicated for me. We also noticed that BQSR was not recommended by nf-core people as well, but none provided a concrete reason beyond "its not useful". As I've also commented on @username_4 post, CRAM didn't work well when using it with IGV, as well as some downstream tools don't work with CRAM.  
  > **username_1**: @username_0 It is interesting that you are criticizing CRAM for slowness in IGV, while you are compressing fastq with spring and you can't even load other alignment formats into IGV. What are you expecting? Also, you don't need to understand WDL. Just search/grep "BaseRecalibrator" or "BQSR" to see if it gets called.  
  > **username_2**: @username_0 sadly, the [GATK best practices](https://tinyurl.com/ykn3bncp), which were updated very recently, still recommend BQSR so it is really surprising that their recommended pipeline doesn't use it. That said, in my place of work, we analyze around 2k human samples a month, and we have found that BQSR often causes us to miss variants. I don't have statistics for this yet so it's anecdotal, but there is strong evidence that BQSR not only isn't particularly helpful today, but can be actively harmful. And not only with small panels, this also happens with whole exome sequencing samples.  
  > **username_1**: @username_2 That page is referencing their older pipelines, which do use BQSR. The WDL file in my link is two years old. It is the latest I can find. IMO, BQSR has been hurting for 10 years. Well, better late than never.  
  > **username_3**: you focus on BAM but what would you suggest for FASTQ if not DRAGEN ORA reference based approach? Arguably long term of storage of FASTQ is more important since BAMs can be regenerated  
  > **username_1**: @username_3 See new edits. I see the value in reference-free algorithms such as spring and dsrc. But if you already require a reference, it is better to keep CRAM than ORA.  
  > **username_2**: For what it's worth, petagene's compatibility layer lets both IGV and JBrowse work out of the box with peta-compressed bam files. You don't even need to use their IGV implementation, vanilla IGV works fine as long as you are using petagene's trick with the symlinks I mentioned in my answer.  
  > **username_1**: @username_3 You can align raw reads with STAR or any local aligner and then create a CRAM. I guess that will give you a smaller file than BAM.  
  > **username_0**: Hay all, thanks for all your comments! I will echo @username_2 and encourage people to share their findings w.r.t. to BQSR not being very helpful. We have been using MarkDuplicatesSpark and BQSR pipeline. I'm planning on dropping the BQSR steps and using samtools markdup -s instead of MarkDuplicatesSpark (Mainly for optimization). And yes, perhaps the lackluster size reduction could have been due to BQSR in data. We also have old NGS data sequenced from 2016 onwards, so some samples might use unbinned quality scores.  
  
  ---
  
  ### Answer by username_4 on 2023-04-11. Score: 5
  
  I'd recommend reconsidering CRAM. It's an open standard that is most likely to still be accessible many years in the future. Regarding your point about lossy compression...
  
  > Can't use lossy [CRAM] compression as these BAM files might be subject to future downstream analysis (Ex. CNV calling, variant calling or something else).
  
  By default, CRAM compression is not lossy.
  
  CRAM does, however, have an optional archive settings mode (`samtools view --output-fmt cram --output-fmt-option archive ...`) which *is* a lossy compression, doing things like removing read names, removing additional accessory fields, and additional compression of quality scores. In all cases, the base sequence of the reads is preserved.
  
  <https://www.htslib.org/doc/samtools.html>
  
  
  ---
  
  ### Answer by username_3 on 2023-04-11. Score: 3
  
  An alternative to the good option of PetaGene that @username_2 mentioned and also not open source is the [DRAGEN ORA compression](https://developer.illumina.com/news-updates/introducing-dragen-original-read-archive-ora). This is loseless reference based compression and can be ~5 x smaller over fastq.gz, but it's only for fastq files and only those generated by Illumina machines (and will be easiest within the DRAGEN ecosystem).
  
  Another point to consider is where you store the data. Even with 'standard' compression you might get more benefits moving archival data to a system like [AWS S3 Glacier](https://en.wikipedia.org/wiki/Amazon_S3_Glacier), depending on your current infastructure setup of course
  
  > **Comments:**
  > **username_5**: This is exactly what I do. I archive FASTQ.GZ (with added content-md5 checksum) to S3. I also archive processed outputs to a different location that is less fault tolerant than S3 as those files can be recreated.  
  > **username_0**: We did check this out, but we do not sequence the NGS data ourselves. We ask external labs to perform the sequencing and they share the fastq.gz files with us. What is the approximate pricing for this? I can't clearly identify where those details are!  
  
  ---
  
  ### Answer by username_2 on 2023-04-11. Score: 3
  
  The only free and open source tool I know that can help is [`zstd`](https://en.wikipedia.org/wiki/Zstd). Their github repository's README describes it as:
  
  > Zstandard, or zstd as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios. It's backed by a very fast entropy stage, provided by Huff0 and FSE library.
  
  We read [this blog post](http://www.bioinformaticszen.com/post/use-zstd-for-raw-fastq/) on using it to compress fastq files and were intrigued so we ran some tests. We couldn't reproduce the same level of compression that the article claims but that might be due to different input files or the way an index was/wasn't created.
  
  Even in our tests though, we did see that it provides better compression than gzip. Here are the specific values for one of the tests, run using an R1 file of a human WGS run:
  
  | Uncompressed fastq | bgzip-compressed | zstd-compressed |
  | --- | --- | --- |
  | 179G | 58G | 47G |
  
  This was done using `-z 19`, to set the compression level to 19 (we found this to be the best tradeoff between size and time to compress). We also saw that using a human genome fasta file to create a compression index, and then using that when compressing human fastq files, actually resulted in worse performance (larger compressed file) than not using the index. But it is entirely possible that we just did it wrong, we didn't spend a lot of time on it.
  
  So you might want to investigate `zstd` as well, maybe you can get better performance if you play around with the indexing. On the plus side, it is free
  and open source. On the other hand, it won't do much with bam or cram files.
  
  > **Comments:**
  > **username_0**: Thanks for the input. I'm pretty sure that with Spring, the output size might be around 20-30 GB, if not slightly lesser. Granted, you would have to compress / decompress the data, but the final file size would be pretty small. We have compressed WGS samples with spring, it goes from total 50GB of gz files to around 15-20 GB in final spring file.  
  > **username_6**: Zstd (and Zpaq and any other general compression tool) deliver relatively bad performance on SAM/BAM files. This is because they are either LZ-based or otherwise view the text as being "one long text", whereas SAM/BAM is anything but that.   FYI, this opinion is based on fairly thorough testing.  
  
  ---
  
  ### Answer by username_2 on 2023-04-11. Score: 3
  
  You might want to consider [PetaGene](https://www.petagene.com/). This is a paid tool, and not open source, but it is something we use at work and it is really good! It can do lossless compression of gzip-compressed fastq files, bam files, and even cram files, and achieves really good size reduction (image from the PetaGene webpage):
  
  [![petagene compression performance bar graph](https://i.sstatic.net/8JqUZ.png)](https://i.sstatic.net/8JqUZ.png)
  
  In my own testing, using files of different sizes, I get an average size reduction of 62.1% (2.6 x smaller) for fastq.gz files (compared to the gzipped fastq file, not compared to the original, uncompressed fastq file) and an average reduction of 82.1% (5.6 x smaller) for bam files. I didn't test cram files.
  
  If you have the budget for a paid service, these guys do good work! The greatest advantage, apart from the very significant size reduction, is that they also provide a system that allows you to use the compressed files natively. So you can feed the petagene-compressed bam file directly to bwa for example. They create special symlinks (`foo.fastq.gz -> foo.fastq.petacompressed`) and you can then use these symlinks and petagene handles the decompression on the fly.
  
  ---
  
  The [Wikipedia page on CRAM](https://en.wikipedia.org/wiki/CRAM_(file_format)) has a nice table with a few alternatives to CRAM which I reproduce below. I haven't tried any except PetaGene though so cannot comment on their performance:
  
  | Name | Key advantage over CRAM | Repo |
  | --- | --- | --- |
  | Deez | None. An early approach to SAM compression, introduced 2014. | <https://github.com/sfu-compbio/deez> |
  | Genozip | Significantly better compression, but slower random access | <https://github.com/divonlan/genozip> |
  | GenomSys | Utilizes ISO-standard MPEG-G, but compression inferior to CRAM 3.1 | Source code not publicly available |
  | PetaSuite | Virtual filesystem interface | Source code not publicly available |
  
  > **Comments:**
  > **username_0**: We considered this software, but sadly we don't have the budget for this. And yes, on-the-fly decompression is a great feature to have, makes storing all data in this format extremely viable. Right now, we get decent compression with spring, but it needs lots of de/compression time, not to mention the CPU resources it guzzles during the process lol  
  > **username_1**: @username_2 genozip is quality work and is very impressive. Its full source code is also on github. If genozip goes under some day (I hope not!), you may create a fork and continue to use their format.  
  > **username_3**: @username_1 ~username_2 - can you share any further info on how PetaGene achieves compression and the critique mentioned of "special treatment of the OQ tag generated by GATK BQSR"? I know it's closed source but trying to understand conceptually how their compression works  
  > **username_2**: Not I, @username_3, sorry. I'm sure username_1 will know far more, but I have no idea and wasn't even aware there was any special treatment before reading their answer. I know at least one person who used to be an engineer working for them is active on the site. If there's something that can be shared, maybe they will answer if you post a question.  
  
  ---
  
  