Answers.Body: <p>Based on conversation in the comments.</p>
  <p><strong>CLR is intended for compositional data. Distance data are not compositional, and compositions are (generally) not symmetric. So you should perform your normalization differently.</strong></p>
  <ol>
  <li>If you want to transform your data for normalization purposes on the features themselves (in your case, the organismal relative abundances), you should do it on the compositional input from which you generate your distances. This is the intended application for CLR. For a reference on normalization approaches for compositional data (including CLR), see <a href="https://www.nature.com/articles/s41522-020-00160-w" rel="nofollow noreferrer">here</a>.</li>
  <li>If you want to transform your distance matrix (e.g. the relationships between samples, irrespective of the underlying relative abundances), then you should use <a href="https://forum.qiime2.org/t/transforms-for-correcting-negative-eigenvalues-for-pcoa-on-semi-metric-distances/17863" rel="nofollow noreferrer">one of the recommended methods</a> for transforming distance data.</li>
  </ol>
  <p>Note that the applications are very different. Application (1) is transforming the underlying organismal frequency data, such that all of the relative abundances are on the same scale. This will make all downstream analyses more stable / interpretable, including ordination analysis.</p>
  <p>Application (2) is saying basically, &quot;my ordination / PCoA on my set of samples <em>still</em> looks weird, even after all that normalization I did on my relative abundances. I want to fix it.&quot; Examples of &quot;looks weird&quot; given include negative eigenvalues.</p>
  <p>You can't substitute (2) for (1). They are working with different data and have different consequences, though both can lead similar issues in your outputs and ordination plots</p>
  <p>It seems like your fundamental problem is that qiime2 outputs are difficult to work with. If that is the case, I would suggest asking a separate question about how to handle qiime2 outputs (or contacting the qiime2 authors about how they do compositional normalization).</p>
  <p>It looks like the qiime2 forum <a href="https://forum.qiime2.org/t/normalization-of-feature-table/13184/5" rel="nofollow noreferrer">already has questions about the right way to apply CLR or related transformations in their workflow</a>. <a href="https://forum.qiime2.org/t/normalisation-of-features/10715/3" rel="nofollow noreferrer">Here is another one</a>. I would suggest following those recommendations and spending some quality time with the tutorials and user forum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T17:34:14.410Z
Answers.LastActivityDate: 2023-06-15T17:34:14.410Z
Answers.UserDisplayName: Maximilian Press
Body: <p>I am trying to transform a distance matrix from the qiime2 pathway using a centred log ratio in R. I uploaded the matrix directly from a qiime2 output file, and have converted to a data matrix and have checked that it is symmetrical.</p>
  <pre class="lang-r prettyprint-override"><code>structure(list(X1 = c(0, 0.2177609998, 0.2133361674, 0.1549136105, 0.1400395799), X11 = c(0.2177609998, 0, 0.07805820645, 0.1418994689, 0.1934668819), X12 = c(0.2133361674, 0.07805820645, 0, 0.1475390242, 0.1857477705), X13 = c(0.1549136105, 0.1418994689, 0.1475390242, 0, 0.1046740994), X14 = c(0.1400395799, 0.1934668819, 0.1857477705, 0.1046740994, 0)), row.names = c(&quot;X1&quot;, &quot;X11&quot;, &quot;X12&quot;, &quot;X13&quot;, &quot;X14&quot;), class = &quot;data.frame&quot;)
  dm &lt;- read.csv(file = &quot;dm.csv&quot;, header = TRUE, row.names = 1)
  isSymmetric(as.matrix(dm)) [1] TRUE
  dm_matrix &lt;- (data.matrix(dm, rownames.force = NA))
  </code></pre>
  <p>I then tried to transform the dataset using the <code>dm_matrix_clr &lt;- data.frame(clr(dm_matrix))</code> function from the 'compositions' package. This produced an output, however it was not symmetrical. From my understanding, the transformed distance matrix should be symmetric like the input matrix. The same numbers on opposing sides of the diagonal are being transformed, but are resulting in different outputs.</p>
  <p><a href="https://i.sstatic.net/ENWBr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ENWBr.png" alt="transformed distance matrix" /></a></p>
  <p>Any help on how resolve this problem so I have a symmetrical transformed distance matrix would be greatly appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-13T05:45:15.427Z
Comments.Text: as i understand it, the transformation assumes compositional data (columns sum to 1). Distances are not compositional. Where did you get the idea for this transformation? Can you link anything? Not really familiar with qiime or surrounding workflows.
Comments.UserDisplayName: Maximilian Press
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-13T13:09:26.280Z
Comments.Text: Here's a paper with a good explanation: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6755255/
  "All components in a composition are mutually dependent features that cannot be understood in isolation. Therefore, any analysis of individual components is done with respect to a reference. This reference transforms each sample into an unbounded space where any statistical method can be used. The centered log-ratio (clr) transformation uses the geometric mean of the sample vector as the reference"
Comments.UserDisplayName: shannonevelyne
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-13T19:56:30.787Z
Comments.Text: that explains why you would apply `clr` to the original compositional data. However it does not follow that `clr` is applicable to distance data derived from compositional data. (distances are not compositions.) Why do you need to transform the distances at all once you have derived them?
Comments.UserDisplayName: Maximilian Press
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-14T14:13:02.663Z
Comments.Text: I need to transform the distances to make the PCoA plot. The way the qiime2 workflow is set up you create the distance matrix file from a feature-table.biom file that is zipped as a .qza file. This makes the actual feature table hard to manipulate, and I have been having trouble exporting that, transforming it, then re importing it into qiime2.
Comments.UserDisplayName: shannonevelyne
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-14T14:25:52.337Z
Comments.Text: https://github.com/LangilleLab/microbiome_helper/wiki/Amplicon-SOP-v2-(qiime2-2022.11)
  
  Here is the workflow I am using in qiime2. The problem is occurring at step 8. As you can see, the deblur_table_final.qza is the input for the diversity metric step (this calculates a whole bunch of diversity tables and graphs, including the beta diversity PcoA plot I want). But as I mentioned before, this file is hard to manipulate & apply the clr() function to. An output of this step is a distance matrix, and I was told on another forum it was possible to transform the distance matrix in R.
Comments.UserDisplayName: shannonevelyne
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-14T16:27:11.780Z
Comments.Text: I am still not clear on why you need to transform the distances (there is no mention of that in the linked doc?), but if you did you could use some of the suggestions from the qiime2 team [here](https://forum.qiime2.org/t/transforms-for-correcting-negative-eigenvalues-for-pcoa-on-semi-metric-distances/17863) (**note these do not include CLR**). I would suggest figuring out how to apply transformations to the feature table (for which you should feel free to post questions here of course!), not transforming the distance matrix, or transforming with one of the recommended methods from that post.
Comments.UserDisplayName: Maximilian Press
Tags: <r>
Title: Why is my centred log ratio transformation on a distance matrix in R not symmetrical?
ThreadText: # Why is my centred log ratio transformation on a distance matrix in R not symmetrical?
  
  (Asked by: username_0 on 2023-06-12)
  
  I am trying to transform a distance matrix from the qiime2 pathway using a centred log ratio in R. I uploaded the matrix directly from a qiime2 output file, and have converted to a data matrix and have checked that it is symmetrical.
  
  ```r
  structure(list(X1 = c(0, 0.2177609998, 0.2133361674, 0.1549136105, 0.1400395799), X11 = c(0.2177609998, 0, 0.07805820645, 0.1418994689, 0.1934668819), X12 = c(0.2133361674, 0.07805820645, 0, 0.1475390242, 0.1857477705), X13 = c(0.1549136105, 0.1418994689, 0.1475390242, 0, 0.1046740994), X14 = c(0.1400395799, 0.1934668819, 0.1857477705, 0.1046740994, 0)), row.names = c("X1", "X11", "X12", "X13", "X14"), class = "data.frame")
  dm <- read.csv(file = "dm.csv", header = TRUE, row.names = 1)
  isSymmetric(as.matrix(dm)) [1] TRUE
  dm_matrix <- (data.matrix(dm, rownames.force = NA))
  
  ```
  
  I then tried to transform the dataset using the `dm_matrix_clr <- data.frame(clr(dm_matrix))` function from the 'compositions' package. This produced an output, however it was not symmetrical. From my understanding, the transformed distance matrix should be symmetric like the input matrix. The same numbers on opposing sides of the diagonal are being transformed, but are resulting in different outputs.
  
  [![transformed distance matrix](https://i.sstatic.net/ENWBr.png)](https://i.sstatic.net/ENWBr.png)
  
  Any help on how resolve this problem so I have a symmetrical transformed distance matrix would be greatly appreciated.
  
  **Tags:** r
  
  > **Comments:**
  > **username_1**: as i understand it, the transformation assumes compositional data (columns sum to 1). Distances are not compositional. Where did you get the idea for this transformation? Can you link anything? Not really familiar with qiime or surrounding workflows.  
  > **username_0**: Here's a paper with a good explanation: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6755255/ "All components in a composition are mutually dependent features that cannot be understood in isolation. Therefore, any analysis of individual components is done with respect to a reference. This reference transforms each sample into an unbounded space where any statistical method can be used. The centered log-ratio (clr) transformation uses the geometric mean of the sample vector as the reference"  
  > **username_1**: that explains why you would apply `clr` to the original compositional data. However it does not follow that `clr` is applicable to distance data derived from compositional data. (distances are not compositions.) Why do you need to transform the distances at all once you have derived them?  
  > **username_0**: I need to transform the distances to make the PCoA plot. The way the qiime2 workflow is set up you create the distance matrix file from a feature-table.biom file that is zipped as a .qza file. This makes the actual feature table hard to manipulate, and I have been having trouble exporting that, transforming it, then re importing it into qiime2.  
  > **username_0**: https://github.com/LangilleLab/microbiome_helper/wiki/Amplicon-SOP-v2-(qiime2-2022.11)  Here is the workflow I am using in qiime2. The problem is occurring at step 8. As you can see, the deblur_table_final.qza is the input for the diversity metric step (this calculates a whole bunch of diversity tables and graphs, including the beta diversity PcoA plot I want). But as I mentioned before, this file is hard to manipulate & apply the clr() function to. An output of this step is a distance matrix, and I was told on another forum it was possible to transform the distance matrix in R.  
  > **username_1**: I am still not clear on why you need to transform the distances (there is no mention of that in the linked doc?), but if you did you could use some of the suggestions from the qiime2 team [here](https://forum.qiime2.org/t/transforms-for-correcting-negative-eigenvalues-for-pcoa-on-semi-metric-distances/17863) (**note these do not include CLR**). I would suggest figuring out how to apply transformations to the feature table (for which you should feel free to post questions here of course!), not transforming the distance matrix, or transforming with one of the recommended methods from that post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-15. Score: 1
  
  Based on conversation in the comments.
  
  **CLR is intended for compositional data. Distance data are not compositional, and compositions are (generally) not symmetric. So you should perform your normalization differently.**
  
  1. If you want to transform your data for normalization purposes on the features themselves (in your case, the organismal relative abundances), you should do it on the compositional input from which you generate your distances. This is the intended application for CLR. For a reference on normalization approaches for compositional data (including CLR), see [here](https://www.nature.com/articles/s41522-020-00160-w).
  2. If you want to transform your distance matrix (e.g. the relationships between samples, irrespective of the underlying relative abundances), then you should use [one of the recommended methods](https://forum.qiime2.org/t/transforms-for-correcting-negative-eigenvalues-for-pcoa-on-semi-metric-distances/17863) for transforming distance data.
  
  Note that the applications are very different. Application (1) is transforming the underlying organismal frequency data, such that all of the relative abundances are on the same scale. This will make all downstream analyses more stable / interpretable, including ordination analysis.
  
  Application (2) is saying basically, "my ordination / PCoA on my set of samples *still* looks weird, even after all that normalization I did on my relative abundances. I want to fix it." Examples of "looks weird" given include negative eigenvalues.
  
  You can't substitute (2) for (1). They are working with different data and have different consequences, though both can lead similar issues in your outputs and ordination plots
  
  It seems like your fundamental problem is that qiime2 outputs are difficult to work with. If that is the case, I would suggest asking a separate question about how to handle qiime2 outputs (or contacting the qiime2 authors about how they do compositional normalization).
  
  It looks like the qiime2 forum [already has questions about the right way to apply CLR or related transformations in their workflow](https://forum.qiime2.org/t/normalization-of-feature-table/13184/5). [Here is another one](https://forum.qiime2.org/t/normalisation-of-features/10715/3). I would suggest following those recommendations and spending some quality time with the tutorials and user forum.
  
  
  ---
  
  