Answers.Body: <p>I am not sure that DADA2 is the right tool for this, though it might help to hear more about your experimental design. I'd recommend reading up on it or looking at materials like <a href="https://web.stanford.edu/class/bios221/Workshops/Lecture2_DADA2_Method.pdf" rel="nofollow noreferrer">this</a>.</p>
  <p>It seems that the mutation rate is relevant to your analysis, such that you are calibrating expected diversity <span class="math-container">$X$</span> based on it. Depending on the expected frequency distribution of your variants, DADA2 might be filtering out most of the variation that you're interested in as sequencing errors.</p>
  <p>See this note on the ASV approach used in DADA2 (<a href="https://academic.oup.com/ismej/article/11/12/2639/7537828" rel="nofollow noreferrer">reference</a>):</p>
  <blockquote>
  <p>ASVs are inferred by a de novo process in which biological sequences are discriminated from errors on the basis of, in part, the expectation that biological sequences are more likely to be repeatedly observed than are error-containing sequences. As a result, ASV inference cannot be performed independently on each read—the smallest unit of data from which ASVs can be inferred is a sample. However, unlike de novo OTUs, ASVs are consistent labels because ASVs represent a biological reality that exists outside of the data being analyzed: the DNA sequence of the assayed organism. Thus, ASVs inferred independently from different studies or different samples can be validly compared.</p>
  </blockquote>
  <p>In brief, you will only be able to infer the presence of mutants that occur at a rate detectably higher than the rate of sequencing error.</p>
  <p>While I am not a DADA2 expert, what DADA2 is expecting is more along the lines of multiple modes in the sequence distribution- each of which is surrounded by a small scattering of sequencing errors resulting in variants nearly identical to the parent mode. It then uses some statistics to evaluate how likely each variant is to be an error off of the nearest mode. So variants that are very close to an existing mode are likely to be lumped into that mode, unless they are sufficiently high abundance to justify establishing a new &quot;species&quot; mode. This has led to a lot of success for DADA2 in accurately identifying variation in samples of relatively diverse samples with somewhat distantly related organisms each at some meaningfully high relative abundance (&gt;=0.1%).</p>
  <p>See for example this figure from the <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4927377/" rel="nofollow noreferrer">original DADA2 paper</a>:</p>
  <p><a href="https://i.sstatic.net/821UGwnT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/821UGwnT.png" alt="enter image description here" /></a></p>
  <p>Every point is a variant identified by DADA2. you'll notice that the variant frequency has to be quite high (~0.1%-1%) to identify distinct variants from sequencing error at a Hamming distance of less than 10 or so from the nearest other variant. Unless you expect your mutant frequencies to be in that range, you are unlikely to detect them. Maybe they've gotten better over the years, but I expect that there are some fundamental statistical issues here for this approach that mean they can't get much better.</p>
  <p>If you really want to use DADA2, any parameter that loosens the statistical thresholds will make it better, but I'd guess you've already tried that. I don't know the QIIME parameters, but if you look at <a href="https://www.bioconductor.org/packages/release/bioc/manuals/dada2/man/dada2.pdf" rel="nofollow noreferrer">dada2 docs</a>, you will see that the <code>dada()</code> function has a number of error parameters that you could play with to try to tune down the probability of errors to increase the likelihood of recovering variants. See the <code>learnErrors()</code> function that you might be able to play with to get reasonable values for this, if you had e.g. a pure reference virus sequenced sample. Note that these functions are for denoising a diverse metagenomic community, not a low-diversity viral mutation experiment! Finally, see <code>setDadaOpt()</code> as ways of globally controlling various thresholds.</p>
  <p>I would recommend doing this within the R package rather than through QIIME, to give you better control. If you can't do that, then maybe re-estimating your <code>FILE-stats.qza</code> or your <a href="https://forum.qiime2.org/t/the-meaning-of-dada2-command-p-min-fold-parent-over-abundance-float/9454/3" rel="nofollow noreferrer"><code>p-min-fold-parent-over-abundance</code> parameter</a> might let you loosen thresholds a bit (though it looks like you are already doing this last somewhat).</p>
  <h2>outside forums</h2>
  <p>see also some <a href="https://forum.qiime2.org/t/up-to-97-of-sequences-being-removed-by-filtering-denoising-in-dada2/12943/2" rel="nofollow noreferrer">similar questions</a> on forums.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-20T21:29:28.770Z
Answers.LastActivityDate: 2024-09-20T21:40:52.997Z
Answers.LastEditDate: 2024-09-20T21:40:52.997Z
Answers.UserDisplayName: Maximilian Press
Body: <p>I am working with sequence reads (acquired through MiSeq platform by Illumina) from a viral inoculum. I have already used FastP to merge paired-end reads and cut 5' and 3' ends to remove adapters and low Phred scores (threshold = 32). The sample should exhibit a genetic diversity that I will label as X. X is defined as the &quot;true theoretical&quot; average sequence identity of all of the reads that should be present in the inoculum (after removing reads with PCR errors and hypermutation and without removing any of the &quot;real&quot; sequences).</p>
  <p>The overall genetic diversity of my inoculum is too high for what I expect given the natural mutation rate of this particular virus (i.e. measured diversity &gt;&gt; X). Previous attempts at denoising or eliminating artefacts still has not lowered the genetic diversity to an acceptable level. Lately, I have been using the DADA2 plugin for QIIME2 (version qiime2-amplicon-2024.5) to denoise my reads, as this has produced the most success in eliminating artefacts. However, it looks like it's been working <em>too well</em> because it has lowered the overall genetic diversity to a level lower than X. This would suggest that DADA2, with my given params/args/options, is not only eliminating artefacts but some of the real sequences as well.</p>
  <p>Is there a set of QIIME DADA2 params/args/options that would help avoid eliminating real sequences during denoising and would achieve a final variant diversity as close to X as possible? Here is the code that I'm using for denoising:</p>
  <pre><code>qiime dada2 denoise-single \
  --i-demultiplexed-seqs FILE.qza \
  --p-trunc-len 0 \
  --p-max-ee 2 \
  --p-pooling-method &quot;independent&quot; \
  --p-chimera-method &quot;none&quot; \
  --p-min-fold-parent-over-abundance 45 \
  --p-allow-one-off \
  --p-n-reads-learn 1000000 \
  --p-no-retain-all-samples \
  --o-table FILE-table.qza \
  --o-representative-sequences FILE-seqs.qza \
  --o-denoising-stats FILE-stats.qza
  </code></pre>
  <p>Thank you for any help you can give me!</p>
  
Tags: <phylogenetics><qiime2>
Title: How to adjust QIIME DADA2 parameters to avoid eliminating real sequences
ThreadText: # How to adjust QIIME DADA2 parameters to avoid eliminating real sequences
  
  (Asked by: username_0 on 2024-09-10)
  
  I am working with sequence reads (acquired through MiSeq platform by Illumina) from a viral inoculum. I have already used FastP to merge paired-end reads and cut 5' and 3' ends to remove adapters and low Phred scores (threshold = 32). The sample should exhibit a genetic diversity that I will label as X. X is defined as the "true theoretical" average sequence identity of all of the reads that should be present in the inoculum (after removing reads with PCR errors and hypermutation and without removing any of the "real" sequences).
  
  The overall genetic diversity of my inoculum is too high for what I expect given the natural mutation rate of this particular virus (i.e. measured diversity >> X). Previous attempts at denoising or eliminating artefacts still has not lowered the genetic diversity to an acceptable level. Lately, I have been using the DADA2 plugin for QIIME2 (version qiime2-amplicon-2024.5) to denoise my reads, as this has produced the most success in eliminating artefacts. However, it looks like it's been working *too well* because it has lowered the overall genetic diversity to a level lower than X. This would suggest that DADA2, with my given params/args/options, is not only eliminating artefacts but some of the real sequences as well.
  
  Is there a set of QIIME DADA2 params/args/options that would help avoid eliminating real sequences during denoising and would achieve a final variant diversity as close to X as possible? Here is the code that I'm using for denoising:
  
  ```
  qiime dada2 denoise-single \
  --i-demultiplexed-seqs FILE.qza \
  --p-trunc-len 0 \
  --p-max-ee 2 \
  --p-pooling-method "independent" \
  --p-chimera-method "none" \
  --p-min-fold-parent-over-abundance 45 \
  --p-allow-one-off \
  --p-n-reads-learn 1000000 \
  --p-no-retain-all-samples \
  --o-table FILE-table.qza \
  --o-representative-sequences FILE-seqs.qza \
  --o-denoising-stats FILE-stats.qza
  
  ```
  
  Thank you for any help you can give me!
  
  **Tags:** phylogenetics, qiime2
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-20. Score: 0
  
  I am not sure that DADA2 is the right tool for this, though it might help to hear more about your experimental design. I'd recommend reading up on it or looking at materials like [this](https://web.stanford.edu/class/bios221/Workshops/Lecture2_DADA2_Method.pdf).
  
  It seems that the mutation rate is relevant to your analysis, such that you are calibrating expected diversity $X$ based on it. Depending on the expected frequency distribution of your variants, DADA2 might be filtering out most of the variation that you're interested in as sequencing errors.
  
  See this note on the ASV approach used in DADA2 ([reference](https://academic.oup.com/ismej/article/11/12/2639/7537828)):
  
  > ASVs are inferred by a de novo process in which biological sequences are discriminated from errors on the basis of, in part, the expectation that biological sequences are more likely to be repeatedly observed than are error-containing sequences. As a result, ASV inference cannot be performed independently on each read—the smallest unit of data from which ASVs can be inferred is a sample. However, unlike de novo OTUs, ASVs are consistent labels because ASVs represent a biological reality that exists outside of the data being analyzed: the DNA sequence of the assayed organism. Thus, ASVs inferred independently from different studies or different samples can be validly compared.
  
  In brief, you will only be able to infer the presence of mutants that occur at a rate detectably higher than the rate of sequencing error.
  
  While I am not a DADA2 expert, what DADA2 is expecting is more along the lines of multiple modes in the sequence distribution- each of which is surrounded by a small scattering of sequencing errors resulting in variants nearly identical to the parent mode. It then uses some statistics to evaluate how likely each variant is to be an error off of the nearest mode. So variants that are very close to an existing mode are likely to be lumped into that mode, unless they are sufficiently high abundance to justify establishing a new "species" mode. This has led to a lot of success for DADA2 in accurately identifying variation in samples of relatively diverse samples with somewhat distantly related organisms each at some meaningfully high relative abundance (>=0.1%).
  
  See for example this figure from the [original DADA2 paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4927377/):
  
  [![enter image description here](https://i.sstatic.net/821UGwnT.png)](https://i.sstatic.net/821UGwnT.png)
  
  Every point is a variant identified by DADA2. you'll notice that the variant frequency has to be quite high (~0.1%-1%) to identify distinct variants from sequencing error at a Hamming distance of less than 10 or so from the nearest other variant. Unless you expect your mutant frequencies to be in that range, you are unlikely to detect them. Maybe they've gotten better over the years, but I expect that there are some fundamental statistical issues here for this approach that mean they can't get much better.
  
  If you really want to use DADA2, any parameter that loosens the statistical thresholds will make it better, but I'd guess you've already tried that. I don't know the QIIME parameters, but if you look at [dada2 docs](https://www.bioconductor.org/packages/release/bioc/manuals/dada2/man/dada2.pdf), you will see that the `dada()` function has a number of error parameters that you could play with to try to tune down the probability of errors to increase the likelihood of recovering variants. See the `learnErrors()` function that you might be able to play with to get reasonable values for this, if you had e.g. a pure reference virus sequenced sample. Note that these functions are for denoising a diverse metagenomic community, not a low-diversity viral mutation experiment! Finally, see `setDadaOpt()` as ways of globally controlling various thresholds.
  
  I would recommend doing this within the R package rather than through QIIME, to give you better control. If you can't do that, then maybe re-estimating your `FILE-stats.qza` or your [`p-min-fold-parent-over-abundance` parameter](https://forum.qiime2.org/t/the-meaning-of-dada2-command-p-min-fold-parent-over-abundance-float/9454/3) might let you loosen thresholds a bit (though it looks like you are already doing this last somewhat).
  
  ## outside forums
  
  see also some [similar questions](https://forum.qiime2.org/t/up-to-97-of-sequences-being-removed-by-filtering-denoising-in-dada2/12943/2) on forums.
  
  
  ---
  
  