Answers.Body: <p>The original answer was &quot;criticised&quot; for being obtuse.</p>
  <p>The basic very simple answer is:</p>
  <ol>
  <li><p>If the data you present in the diagram (alignment) is left EXACTLY in the format used to make the tree - it honestly does not matter whether you have 1 partition or 2 partitions. It makes no practical difference. The posterior likelihood and stuff will change a bit - basically the result is the same. The reason is given in the very long explanation below. I admit 2-partitions looks cooler and there's reason not to do it, computationally it will be a bit slower.
  <strong>Partitions</strong> I am happy to explain why a partition is used  - but thats different answer or question.</p>
  </li>
  <li><p>What must not be done with HIV is changing every '-' to '?' or changing every '-' to 'N'. Whether this is 1-partition or 2-partitions. With higher eukaryotes that is fine and that will work very nicely, but HIV no. So Weins et al (2012) can do this (almost certainly have done), but you can't.</p>
  </li>
  </ol>
  <p>I know it seems a very small change replacing every '-' with a '?' or 'N', but what it represents seismic. I have attempted to explain why it is so important below the line, but it's about indels (insertions/deletions). I recognise my explanation may not be perfect.</p>
  <p><strong>Confusion</strong> The confusing bit is you mention 'NNNNNs' and thats the worry. If NNNNs are changed to '-' thats fine, if they are left as 'NNNNs' that raises the complicated spectre of 'indels' and will get very complicated. It probably can be done but its really complicated.</p>
  <p>To make the point clear if I perform a <code>trimal</code> analysis on my alignment</p>
  <pre><code>trimal -in infile -out outfile -nogap -fasta
  </code></pre>
  <p>It will remove all sites with '-' in them, i.e. it will remove the entire site, not just the '-' character. I think it's really cool. Anyway, the tree from a <code>trimal</code> alignment will be basically the same as '1-partition' or '2-partitions' with all the '-' retained, even though the alignments look very different. That sounds weird but it is a consequence of how Bayes (i.e. Beast) or ML calculate phylogenetic information, which I tried to explain below. HOWEVER, if there is an alignment with 'NNNNs' instead of '-' that will produce a very different tree.</p>
  <p>So for simplicity the answer is that 2-partitions is fine for '-', but not if those '-' are represented as 'NNNNN'. Thus for '-' alignments it looks cool, but makes little difference. 2-partitions can make a HUGE difference with different data particularly with loads and loads of NNNNNs instead of '-'.</p>
  <hr />
  <p><strong>Summary</strong> HIV is a special case because <em>env</em> in particular - and possibly all the HIV genes - are known to contain extensive indels. Indels in <em>env</em> are so prevalent in HIV they can be approximated via distribution (did Ziheng Yang do that?). So the short answer, Weins <em>et al</em> (2012) can do it because they're dealing with higher eukaryotes and genes with negligible indels (in addition they could remove known indel sites), in HIV you would need independent evidence the missing data site was not going to be an indel (i.e. that particular site has never had a known indel in any HIV data set).</p>
  <p>The crux is there is a difference between missing data and absent data. Absent data <em>can be</em> a no-go (below; context depending - some times it makes no difference [immediately below]), it can be detrimental to the phylogenetic signal. Missing data is very interesting.</p>
  <p>Basically, if you are going to leave the missing 2kb gene as '-' ... thats perfectly fine. It will not add any phylogenetic information - but it will not hurt the calculation. If 'N' or '?' are used instead to represent a missing 2Kb locus for HIV instead  of '-' thats different.</p>
  <hr />
  <p>Weins <em>et al</em> (2012) is a very interesting study.</p>
  <p>There are four issues</p>
  <ol>
  <li>Sample skews</li>
  <li>Absent data which causes blockwise deletions in Bayes and ML</li>
  <li>Missing data which <em>might</em> recover phylogenetic information at a given site if correctly annotated.
  4.Quality of data</li>
  </ol>
  <p>The traditional view is that it is better to remove the site/ or sample, if the quality of data is in doubt or for example an entire gene is missing. The traditional view is it is better to be certain of the data regardless of possible risk of a sample skew. However, <em>sample skews</em> are the Achilles heal of population genetics and a lesser extent to phylogenetics. Thats what Weins <em>et al</em> (2012) is concluding, it's really cool. I like it.</p>
  <p>We all know that an analyst can't deliberately cause a sample skew (i.e. you can't omit samples without good reason) - that goes with out saying. Right?</p>
  <p>The context of the question is &quot;if a sample has <em>pol</em> and its other gene wasn't sequenced - do you fill in the absent gene with &quot;Ns&quot; or &quot;?&quot; to trap the phylogenetic information in <em>pol</em> and the samples which have data in the gene/locus/partition with limited coverage. <strong>It's a really good question.</strong> My suspicion for HIV is I'm afraid you probably can't, but that might just be <em>env</em>.</p>
  <p><strong>Absent data</strong>
  Bayesian and ML have no '-' character in their matrix. If an indel is present it will trigger that site to be excluded from carrying any phylogenetic information specifically for Bayes and ML. Thus</p>
  <pre><code>                Site1
  Sample 1         A
  Sample 2         A
  Sample 3         A
  ....
  Sample 9999      -
  Sample 10000     A
  </code></pre>
  <p>Site 1 will not carry any phylogenetic information for Bayes or ML because the single '-' is not represented in the mutation matrix. The entire site for 9999 nucleotides is excluded. A pairwise analysis would be needed, e.g. distance methods.</p>
  <p><strong>Missing data</strong> However &quot;Sample 9999, Site 1&quot; could be represented as 'N' instead of a '-' because that means any nucleotide in the matrix. <em>That works!</em> Site 1 is now phylogenetically informative under Bayes and ML. Hooray! However, are we <em>certain</em> Sample 9999, Site 1 is a nucleotide rather than an indel? If indels are rewritten as nucleotides, i.e. an absent nucleotide is converted to a nucleotide of unknown type ('AGCT'), i.e. missing data - non-nucleotide to nucleotide - thats cheating.</p>
  <p>This creates issues <strong>particularly</strong> for HIV <em>env</em> where there are loads of indels. If an indel was replaced with an 'N' - the analysis might work nicely, but there is real risk it is not being true to the data.</p>
  <hr />
  <p><strong>Possibly explanation</strong> Sample skews in phylogeny are really interesting. What can happen is long internal branches can cause changes to the branching order. However, if those internal branches are shorten by more rigorous taxon sampling then these shortened internal branches can recover the true true. I think it was Jo Felsenstein that discovered this (the guy who invented maximum likelihood as a method for phylogenetics in place of parsimony).</p>
  <p>One explanation of Weins <em>et al</em> (2012) is that the 'missing data' samples are shortening the distance between internal branches - by providing a more rigorous sample - and that is resulting in recovery of the correct tree. So it is probably a result inline with Jo Felsensteins original observation.</p>
  <p>So if <em><strong>pol</strong></em> never has indels - ever - you can do it. This would also help population dynamics and all shorts of calculations. If there's a risk that you are replacing indels with nucleotides it is very much, much better to avoid it.</p>
  <p><strong>Conclusion</strong> Everyone in phylogentics knows the issues of sample skews - a possible sample skew is permitted if the data is genuinely never been produced - it's not in existence (obviously). However, it is not permitted if the data exists but the analyst hasn't included it - for whatever reason. Its also preferred if the quality is in question - its better to err on the side of caution.</p>
  <p>If you fill in an absent gene with &quot;?&quot; or &quot;N&quot; to obtain phylogenetic information from genes that are present for that sample ... that is permitted providing it doesn't change the possible nature of the underlying data. <em>The rationale for doing this is the risk of sample skews.</em></p>
  <hr />
  <p>So would I adopt a &quot;non-traditional&quot; approach? As a general rule for viruses no and this is a particular special case regarding indels.</p>
  <p>If this was a higher eukaryote and like - that critical sample and  <em>that</em> gene - didn't give sequence. This has happened. So would I dump the sample (or in this case the locus) - no I'd caveat the tree and explain and empirically check the assumptions.</p>
  <p>If I was in a situation where 2/3 of the data were missing, for a higher eukaryote (DNA with great proof-reading), personally I would do a full scale investigation, i.e. phylogenies with and without the gene/locus/partition in question. That is pretty much was Weins et al (2012) have done.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-25T13:32:22.573Z
Answers.Text: Thank you very much for your reasoning! My question was mostly dedicated to the whole partitions, not single incomplete sites or indels. My sequences jointly cover only *pol*, but it also has a bunch of indels. I am intent on not including insertions and exchanging deletions with `N`.
Answers.UserDisplayName: Vovin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T17:26:50.030Z
Answers.Text: @M__ I am really thankful for your attention! But please, could you shed light on your final point about indels? Have I understood you correctly that we should exclude all insertion columns and exchange deletions with `N` (I was intent on doing this initially), shouldn't we? Or we should exclude deletion sites as well, or we cannot analyse HIV fragments susceptible to indels with Bayesian MCMC at all?
Answers.UserDisplayName: Vovin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T20:29:54.570Z
Answers.Text: @Vovin I've summarised the basic answer above the question. Please check it and if it still not clear I will attempt another explanation.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-24T21:21:16.600Z
Answers.LastActivityDate: 2023-04-26T21:16:24.553Z
Answers.LastEditDate: 2023-04-26T21:16:24.553Z
Answers.UserDisplayName: M__
Body: <p>I am seeking advice on whether or not to incorporate incomplete data into my <strong>BEAST</strong> analysis.</p>
  <p>I have HIV data consisting of two partitions of the <em>pol</em> gene: one that is 1.2 <em>kb</em> in length and present in all isolates, &amp; another that is 2 <em>kb</em> in length but only present in one-third of the isolates. I am wondering if I should include the second partition in my analysis, despite its incomplete nature.</p>
  <p>I have found some publications discussing the use of incomplete data in phylogenetic analyses, such as <a href="https://doi.org/10.1016/j.ympev.2015.02.002" rel="nofollow noreferrer">Yuchi Zheng &amp; John J. Wiens (2015)</a>, <a href="https://doi.org/10.1016/j.ympev.2013.06.004" rel="nofollow noreferrer">Rasmus Hovmöller et al. (2013)</a>, <a href="https://doi.org/10.1371/journal.pone.0042925" rel="nofollow noreferrer">John J. Wiens &amp; Jonathan Tiu (2012)</a>. However, I remain uncertain about the best course of action.</p>
  <p>If you have experience working with incomplete data in BEAST analyses or any insights on this matter, I would greatly appreciate your advice.</p>
  <p>Thank you in advance for your help.</p>
  <p>Best regards,</p>
  <hr />
  <h2>Update</h2>
  <p>As seen from the answer &amp; comments, there has been a misinterpretation of my partitions' structure &amp; their incompleteness. So hereinbelow is a short description of my data, the subject topic, &amp; a diagram.</p>
  <p>There are 1009 isolates of the <em>pol</em> gene of HIV from the Republic of Armenia. They were sequenced from the start of the gene by two means: Sanger &amp; an NGS; which caused two groups of length: 1.2 kb &amp; &gt; 3 kb, respectively.</p>
  <p>This way the situation may be demonstrated via the next artificial miniature diagram:</p>
  <p><a href="https://i.sstatic.net/f7wnI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/f7wnI.png" alt="diagram with a little situation example" /></a></p>
  <p>Based on the picture like above, I have thought it rational to divide the data into two staple partitions: the first third of <em>pol</em> &amp; its last part. The first partition has all sequences completed, and the second one has mostly full-length complete sequences for a third of isolates and just <code>NNN…NNN</code>s for the other two-thirds.</p>
  <p><code>N</code>s are automatically added at the alignment step. Sanger sequences are just short and end from 1200 to 1500. I currently do not have access to the source data. They were provided to me already built and aligned at the reference.</p>
  <p>In summary, the question is, <strong>&quot;Should only the complete first partition be used in subsequent BEAST analysis or should the incomplete second one be included as well?&quot;</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-26T16:58:04.607Z
Comments.Text: @gringer I have updated the question.
Comments.UserDisplayName: Vovin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-26T19:36:20.893Z
Comments.Text: Thanks, that's a great improvement; much clearer now.
Comments.UserDisplayName: gringer
Tags: <phylogenetics><phylogeny><virus><beast><hiv>
Title: What is the limit of partition incompleteness in Bayesian MCMC?
ThreadText: # What is the limit of partition incompleteness in Bayesian MCMC?
  
  (Asked by: username_0 on 2023-04-24)
  
  I am seeking advice on whether or not to incorporate incomplete data into my **BEAST** analysis.
  
  I have HIV data consisting of two partitions of the *pol* gene: one that is 1.2 *kb* in length and present in all isolates, & another that is 2 *kb* in length but only present in one-third of the isolates. I am wondering if I should include the second partition in my analysis, despite its incomplete nature.
  
  I have found some publications discussing the use of incomplete data in phylogenetic analyses, such as [Yuchi Zheng & John J. Wiens (2015)](https://doi.org/10.1016/j.ympev.2015.02.002), [Rasmus Hovmöller et al. (2013)](https://doi.org/10.1016/j.ympev.2013.06.004), [John J. Wiens & Jonathan Tiu (2012)](https://doi.org/10.1371/journal.pone.0042925). However, I remain uncertain about the best course of action.
  
  If you have experience working with incomplete data in BEAST analyses or any insights on this matter, I would greatly appreciate your advice.
  
  Thank you in advance for your help.
  
  Best regards,
  
  ---
  
  ## Update
  
  As seen from the answer & comments, there has been a misinterpretation of my partitions' structure & their incompleteness. So hereinbelow is a short description of my data, the subject topic, & a diagram.
  
  There are 1009 isolates of the *pol* gene of HIV from the Republic of Armenia. They were sequenced from the start of the gene by two means: Sanger & an NGS; which caused two groups of length: 1.2 kb & > 3 kb, respectively.
  
  This way the situation may be demonstrated via the next artificial miniature diagram:
  
  [![diagram with a little situation example](https://i.sstatic.net/f7wnI.png)](https://i.sstatic.net/f7wnI.png)
  
  Based on the picture like above, I have thought it rational to divide the data into two staple partitions: the first third of *pol* & its last part. The first partition has all sequences completed, and the second one has mostly full-length complete sequences for a third of isolates and just `NNN…NNN`s for the other two-thirds.
  
  `N`s are automatically added at the alignment step. Sanger sequences are just short and end from 1200 to 1500. I currently do not have access to the source data. They were provided to me already built and aligned at the reference.
  
  In summary, the question is, **"Should only the complete first partition be used in subsequent BEAST analysis or should the incomplete second one be included as well?"**
  
  **Tags:** phylogenetics, phylogeny, virus, beast, hiv
  
  > **Comments:**
  > **username_0**: @username_1 I have updated the question.  
  > **username_1**: Thanks, that's a great improvement; much clearer now.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-24. Score: 1
  
  The original answer was "criticised" for being obtuse.
  
  The basic very simple answer is:
  
  1. If the data you present in the diagram (alignment) is left EXACTLY in the format used to make the tree - it honestly does not matter whether you have 1 partition or 2 partitions. It makes no practical difference. The posterior likelihood and stuff will change a bit - basically the result is the same. The reason is given in the very long explanation below. I admit 2-partitions looks cooler and there's reason not to do it, computationally it will be a bit slower.
     **Partitions** I am happy to explain why a partition is used - but thats different answer or question.
  2. What must not be done with HIV is changing every '-' to '?' or changing every '-' to 'N'. Whether this is 1-partition or 2-partitions. With higher eukaryotes that is fine and that will work very nicely, but HIV no. So Weins et al (2012) can do this (almost certainly have done), but you can't.
  
  I know it seems a very small change replacing every '-' with a '?' or 'N', but what it represents seismic. I have attempted to explain why it is so important below the line, but it's about indels (insertions/deletions). I recognise my explanation may not be perfect.
  
  **Confusion** The confusing bit is you mention 'NNNNNs' and thats the worry. If NNNNs are changed to '-' thats fine, if they are left as 'NNNNs' that raises the complicated spectre of 'indels' and will get very complicated. It probably can be done but its really complicated.
  
  To make the point clear if I perform a `trimal` analysis on my alignment
  
  ```
  trimal -in infile -out outfile -nogap -fasta
  
  ```
  
  It will remove all sites with '-' in them, i.e. it will remove the entire site, not just the '-' character. I think it's really cool. Anyway, the tree from a `trimal` alignment will be basically the same as '1-partition' or '2-partitions' with all the '-' retained, even though the alignments look very different. That sounds weird but it is a consequence of how Bayes (i.e. Beast) or ML calculate phylogenetic information, which I tried to explain below. HOWEVER, if there is an alignment with 'NNNNs' instead of '-' that will produce a very different tree.
  
  So for simplicity the answer is that 2-partitions is fine for '-', but not if those '-' are represented as 'NNNNN'. Thus for '-' alignments it looks cool, but makes little difference. 2-partitions can make a HUGE difference with different data particularly with loads and loads of NNNNNs instead of '-'.
  
  ---
  
  **Summary** HIV is a special case because *env* in particular - and possibly all the HIV genes - are known to contain extensive indels. Indels in *env* are so prevalent in HIV they can be approximated via distribution (did Ziheng Yang do that?). So the short answer, Weins *et al* (2012) can do it because they're dealing with higher eukaryotes and genes with negligible indels (in addition they could remove known indel sites), in HIV you would need independent evidence the missing data site was not going to be an indel (i.e. that particular site has never had a known indel in any HIV data set).
  
  The crux is there is a difference between missing data and absent data. Absent data *can be* a no-go (below; context depending - some times it makes no difference [immediately below]), it can be detrimental to the phylogenetic signal. Missing data is very interesting.
  
  Basically, if you are going to leave the missing 2kb gene as '-' ... thats perfectly fine. It will not add any phylogenetic information - but it will not hurt the calculation. If 'N' or '?' are used instead to represent a missing 2Kb locus for HIV instead of '-' thats different.
  
  ---
  
  Weins *et al* (2012) is a very interesting study.
  
  There are four issues
  
  1. Sample skews
  2. Absent data which causes blockwise deletions in Bayes and ML
  3. Missing data which *might* recover phylogenetic information at a given site if correctly annotated.
     4.Quality of data
  
  The traditional view is that it is better to remove the site/ or sample, if the quality of data is in doubt or for example an entire gene is missing. The traditional view is it is better to be certain of the data regardless of possible risk of a sample skew. However, *sample skews* are the Achilles heal of population genetics and a lesser extent to phylogenetics. Thats what Weins *et al* (2012) is concluding, it's really cool. I like it.
  
  We all know that an analyst can't deliberately cause a sample skew (i.e. you can't omit samples without good reason) - that goes with out saying. Right?
  
  The context of the question is "if a sample has *pol* and its other gene wasn't sequenced - do you fill in the absent gene with "Ns" or "?" to trap the phylogenetic information in *pol* and the samples which have data in the gene/locus/partition with limited coverage. **It's a really good question.** My suspicion for HIV is I'm afraid you probably can't, but that might just be *env*.
  
  **Absent data**
  Bayesian and ML have no '-' character in their matrix. If an indel is present it will trigger that site to be excluded from carrying any phylogenetic information specifically for Bayes and ML. Thus
  
  ```
                  Site1
  Sample 1         A
  Sample 2         A
  Sample 3         A
  ....
  Sample 9999      -
  Sample 10000     A
  
  ```
  
  Site 1 will not carry any phylogenetic information for Bayes or ML because the single '-' is not represented in the mutation matrix. The entire site for 9999 nucleotides is excluded. A pairwise analysis would be needed, e.g. distance methods.
  
  **Missing data** However "Sample 9999, Site 1" could be represented as 'N' instead of a '-' because that means any nucleotide in the matrix. *That works!* Site 1 is now phylogenetically informative under Bayes and ML. Hooray! However, are we *certain* Sample 9999, Site 1 is a nucleotide rather than an indel? If indels are rewritten as nucleotides, i.e. an absent nucleotide is converted to a nucleotide of unknown type ('AGCT'), i.e. missing data - non-nucleotide to nucleotide - thats cheating.
  
  This creates issues **particularly** for HIV *env* where there are loads of indels. If an indel was replaced with an 'N' - the analysis might work nicely, but there is real risk it is not being true to the data.
  
  ---
  
  **Possibly explanation** Sample skews in phylogeny are really interesting. What can happen is long internal branches can cause changes to the branching order. However, if those internal branches are shorten by more rigorous taxon sampling then these shortened internal branches can recover the true true. I think it was Jo Felsenstein that discovered this (the guy who invented maximum likelihood as a method for phylogenetics in place of parsimony).
  
  One explanation of Weins *et al* (2012) is that the 'missing data' samples are shortening the distance between internal branches - by providing a more rigorous sample - and that is resulting in recovery of the correct tree. So it is probably a result inline with Jo Felsensteins original observation.
  
  So if ***pol*** never has indels - ever - you can do it. This would also help population dynamics and all shorts of calculations. If there's a risk that you are replacing indels with nucleotides it is very much, much better to avoid it.
  
  **Conclusion** Everyone in phylogentics knows the issues of sample skews - a possible sample skew is permitted if the data is genuinely never been produced - it's not in existence (obviously). However, it is not permitted if the data exists but the analyst hasn't included it - for whatever reason. Its also preferred if the quality is in question - its better to err on the side of caution.
  
  If you fill in an absent gene with "?" or "N" to obtain phylogenetic information from genes that are present for that sample ... that is permitted providing it doesn't change the possible nature of the underlying data. *The rationale for doing this is the risk of sample skews.*
  
  ---
  
  So would I adopt a "non-traditional" approach? As a general rule for viruses no and this is a particular special case regarding indels.
  
  If this was a higher eukaryote and like - that critical sample and *that* gene - didn't give sequence. This has happened. So would I dump the sample (or in this case the locus) - no I'd caveat the tree and explain and empirically check the assumptions.
  
  If I was in a situation where 2/3 of the data were missing, for a higher eukaryote (DNA with great proof-reading), personally I would do a full scale investigation, i.e. phylogenies with and without the gene/locus/partition in question. That is pretty much was Weins et al (2012) have done.
  
  > **Comments:**
  > **username_0**: Thank you very much for your reasoning! My question was mostly dedicated to the whole partitions, not single incomplete sites or indels. My sequences jointly cover only *pol*, but it also has a bunch of indels. I am intent on not including insertions and exchanging deletions with `N`.  
  > **username_0**: @username_2 I am really thankful for your attention! But please, could you shed light on your final point about indels? Have I understood you correctly that we should exclude all insertion columns and exchange deletions with `N` (I was intent on doing this initially), shouldn't we? Or we should exclude deletion sites as well, or we cannot analyse HIV fragments susceptible to indels with Bayesian MCMC at all?  
  > **username_2**: @username_0 I've summarised the basic answer above the question. Please check it and if it still not clear I will attempt another explanation.  
  
  ---
  
  