Answers.Body: <p>Good question. You have performed the first step via unsupervised learning and AU bootstraps give great results, orthodox bootstraps give no significant (or very few) clusters. The contrast between these approaches is unusual because they should be somewhere close to each other. If you can resolve that, your analysis is fine. So at present IMO the unsupervised learning needs further investigation. 
  ... deleted </p>
  
  <p>... <strong>antibodies</strong> ... oh I see. Trees, like you've done, don't work here because of the cross-reactivity creates network phylogenies. This is the antonym of bifurcating trees. If someone has a robust bifurcating phylogeny with antibody data and gets good bootstraps .. utterly amazing. However, if you subject your data to a network analysis you'll find connections all over the place - in practical terms this really messes up bootstraps. Thats the likely reason for you results. There is no point doing a complex bifurcating tree analysis because the antibody cross-reactivity will mess it up. k-means has been implemented as unsupervised learning in these instances and would provide the starting point in using clustering to define a supervised learning problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T01:15:49.207Z
Answers.Text: could you briefly explain how K-means clustering would work here? would I simple choose k=2 (cases vs controls) and show that cases and controls do not cluster into 2 groups? Could I still bootstrap the result to give significance?
Answers.UserDisplayName: reubenmcg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T12:47:42.807Z
Answers.Text: No not in this instance ... with "traditional" k-means is you have to set the groups a priori 2,3,4,5 etc..... and the trick is defining which a priori grouping provides the "best fit" (I forget the criterion). The tree will provide some indication of the max group size. K-means will then assign all you data into the designated number of clusters as best it can. K-means is respectable - albeit it doesn't sound cool. I think its because it minimises false positives and robust. The study I was involved with (malaria) used k-means as the singular proof :-O
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T15:22:10.170Z
Answers.Text: In this study, the network phylogeny defined the groupings (it couldn't assign significance to it) and these groupings were explored via K-means. Trees (network trees) were there just because I like them, all the tree was saying that was statically significant was "these things cross-react" (hardly revolutionary). Otherwise it is a bit hard to describe k-means within a comments section, but let me know if you get the gist.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T15:22:32.717Z
Answers.Text: I am doing supervised learning on quite similar data to yours at present but for viruses. However, this is a bit too close my current work to be providing detailed technical advice.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-16T00:16:45.440Z
Answers.LastActivityDate: 2019-04-16T19:22:25.900Z
Answers.LastEditDate: 2019-04-16T19:22:25.900Z
Answers.UserDisplayName: M__
Answers.Body: <p>I've got it. 
  The question you are asking is homogeneity between you target group and your general population of patients. What you want to show is "complete" homogeneity between the two groups regardless of any sub-sampling, ie. if there is groups within groups that might break your hypothesis.
  K-means is perfect here. The issue with K-means is designating the number of groups, but here thats irrelevant because its not the question you are asking. You are asking homogeneity and if your hypothesis is true no matter how many groups you designate for K-means, e.g. 2,3,4,5,6,7,8,9,10 (or maybe a more reduced sampling) homogeneity between target (treatment?) and general population is observed. This would therefore be a definitive analysing ASSUMING that you don't encounter a group where there is exclusively one population and the other is absent. Obviously the number of K-mean groups can't exceed the number of treated/target patients.</p>
  
  <p>It would be a really good application of k-means because it avoids the question, 
  So why was that a priori selected? </p>
  
  <p>Personally I would first present your preferred a priori group size and then say same result for all group sizes. Its simple, recognised and neat and easy for a reviewer to follow. There are ML solutions to K-means, but I think "keep it simple".</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-18T02:06:42.813Z
Answers.Text: This makes allot of sense, and just knowing to google homogeneity has led me down what I think is the correct path, I am now struggling to find R packages that let me do this with explanations on how to sue them. I think clValid will work
Answers.UserDisplayName: reubenmcg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-18T16:54:18.130Z
Answers.Text: There is a direct function called 'kmeans' in R, but its not an approach I'd use unless I was reiterating within a data pipeline (R is great for automation). For this a general stats package SPSS or Stata is easier. ML k-means are in Sci-kit learn (Python), but this is really complicated and lots of things can go wrong. Glad this helped.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-17T17:27:01.810Z
Answers.LastActivityDate: 2019-04-17T17:27:01.810Z
Answers.UserDisplayName: M__
Body: <p>I have data that includes 'cases' and 'controls' and have carried out hierarchical clustering. </p>
  
  <p>I used the <code>pvclust</code> package in R to bootstrap the results and significant branches are highlighted with red rectangles (based on au>0.95):</p>
  
  <p><a href="https://i.sstatic.net/C3kwC.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C3kwC.jpg" alt="enter image description here"></a></p>
  
  <p>What is clear is that no clustering occurs that separates 'cases' and 'controls', and this is in fact what we expected and want to show. We want to show that the measured variable does not distinguish between cases and controls.</p>
  
  <ol>
  <li>List <code>itemApart</code> from saying visually no clear clusters emerge that distinguishes between cases and controls are there any objective measures that can be used to say no significant clustering occurs between two groups?</li>
  <li>One observation I have here is that the AU values and the BP values are very different, even though both p-values should be interpreted in a similar fashion, am I missing something?</li>
  <li><strong>Perhaps <code>pvclust</code> (bootstrapping) is not the right option here, is there a better way of showing quantifiably that no significant clustering occurs between two groups? Perhaps some kind of supervised clustering (I am not sure what this even means in this context)?</strong></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-15T11:56:43.677Z
Comments.Text: I am unfamiliar with this package but some red flags: 1) you use correlation as a distance measure, it makes more sense as a similarity measure, 2) I don't see a cutoff in the tree to determine what your actual clustering is. This might give a view where clusters do emerge with distinct case/control patterns. edit: I see the red brackets on the bottom now, are these your final cluster labels?
Comments.UserDisplayName: Pallie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-15T12:23:48.360Z
Comments.Text: Can you explain more about your data please? Is this gene expression? Or something else?
Comments.UserDisplayName: benn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-15T22:18:36.520Z
Comments.Text: @Pallie, I have cut off a section of the picture which has patient ID's in it so there is a blank white space which obscures the red boxes sorry. If you look at the red boxes on the dendrogram they highlight branches with AU values >95 which I interpret as meaning these are the statistically significant clusters.
  
  Benn, this I not gene expression, it is a measure of antibodies to about 100 bacterial strains in cases vs controls. 
  
  Any advice on a better measure to use perhaps to show the measure does not cluster patients and controls? Thank you both
Comments.UserDisplayName: reubenmcg
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-16T08:44:08.880Z
Comments.Text: I'm not sure doing clustering and then trying to prove that the clustering is significant is the right approach, it is very heavily dependent on the clustering algorithm you used. Can't you just do an ANOVA between case/control to prove there is a/no significant difference?
Comments.UserDisplayName: Pallie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-16T19:13:37.367Z
Comments.Text: ANOVA requires normalised data and its unlikely to be Gaussian (it sounds a bit odd saying "unlikely to be normal).
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-16T19:19:51.497Z
Comments.Text: The take-home lesson from this question is always ask what data it is. I'd assumed it was some sort of gene expression, antibody data is radically different and explains the results better. The problem is that IgG will be capable of binding to some degree with any strain in the analysis.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-16T20:56:32.283Z
Comments.Text: Thanks, yes lesson learned, sorry for the confusion, allot to learn.
Comments.UserDisplayName: reubenmcg
Tags: <r><clustering>
Title: Interpretation of hierarchical clustering with bootstrapping
ThreadText: # Interpretation of hierarchical clustering with bootstrapping
  
  (Asked by: username_0 on 2019-04-14)
  
  I have data that includes 'cases' and 'controls' and have carried out hierarchical clustering.
  
  I used the `pvclust` package in R to bootstrap the results and significant branches are highlighted with red rectangles (based on au>0.95):
  
  [![enter image description here](https://i.sstatic.net/C3kwC.jpg)](https://i.sstatic.net/C3kwC.jpg)
  
  What is clear is that no clustering occurs that separates 'cases' and 'controls', and this is in fact what we expected and want to show. We want to show that the measured variable does not distinguish between cases and controls.
  
  1. List `itemApart` from saying visually no clear clusters emerge that distinguishes between cases and controls are there any objective measures that can be used to say no significant clustering occurs between two groups?
  2. One observation I have here is that the AU values and the BP values are very different, even though both p-values should be interpreted in a similar fashion, am I missing something?
  3. **Perhaps `pvclust` (bootstrapping) is not the right option here, is there a better way of showing quantifiably that no significant clustering occurs between two groups? Perhaps some kind of supervised clustering (I am not sure what this even means in this context)?**
  
  **Tags:** r, clustering
  
  > **Comments:**
  > **username_1**: I am unfamiliar with this package but some red flags: 1) you use correlation as a distance measure, it makes more sense as a similarity measure, 2) I don't see a cutoff in the tree to determine what your actual clustering is. This might give a view where clusters do emerge with distinct case/control patterns. edit: I see the red brackets on the bottom now, are these your final cluster labels?  
  > **username_2**: Can you explain more about your data please? Is this gene expression? Or something else?  
  > **username_0**: @username_1, I have cut off a section of the picture which has patient ID's in it so there is a blank white space which obscures the red boxes sorry. If you look at the red boxes on the dendrogram they highlight branches with AU values >95 which I interpret as meaning these are the statistically significant clusters.  username_2, this I not gene expression, it is a measure of antibodies to about 100 bacterial strains in cases vs controls.   Any advice on a better measure to use perhaps to show the measure does not cluster patients and controls? Thank you both  
  > **username_1**: I'm not sure doing clustering and then trying to prove that the clustering is significant is the right approach, it is very heavily dependent on the clustering algorithm you used. Can't you just do an ANOVA between case/control to prove there is a/no significant difference?  
  > **username_3**: ANOVA requires normalised data and its unlikely to be Gaussian (it sounds a bit odd saying "unlikely to be normal).  
  > **username_3**: The take-home lesson from this question is always ask what data it is. I'd assumed it was some sort of gene expression, antibody data is radically different and explains the results better. The problem is that IgG will be capable of binding to some degree with any strain in the analysis.  
  > **username_0**: Thanks, yes lesson learned, sorry for the confusion, allot to learn.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-04-16. Score: 1 | Accepted answer
  
  Good question. You have performed the first step via unsupervised learning and AU bootstraps give great results, orthodox bootstraps give no significant (or very few) clusters. The contrast between these approaches is unusual because they should be somewhere close to each other. If you can resolve that, your analysis is fine. So at present IMO the unsupervised learning needs further investigation.
  ... deleted
  
  ... **antibodies** ... oh I see. Trees, like you've done, don't work here because of the cross-reactivity creates network phylogenies. This is the antonym of bifurcating trees. If someone has a robust bifurcating phylogeny with antibody data and gets good bootstraps .. utterly amazing. However, if you subject your data to a network analysis you'll find connections all over the place - in practical terms this really messes up bootstraps. Thats the likely reason for you results. There is no point doing a complex bifurcating tree analysis because the antibody cross-reactivity will mess it up. k-means has been implemented as unsupervised learning in these instances and would provide the starting point in using clustering to define a supervised learning problem.
  
  > **Comments:**
  > **username_0**: could you briefly explain how K-means clustering would work here? would I simple choose k=2 (cases vs controls) and show that cases and controls do not cluster into 2 groups? Could I still bootstrap the result to give significance?  
  > **username_3**: No not in this instance ... with "traditional" k-means is you have to set the groups a priori 2,3,4,5 etc..... and the trick is defining which a priori grouping provides the "best fit" (I forget the criterion). The tree will provide some indication of the max group size. K-means will then assign all you data into the designated number of clusters as best it can. K-means is respectable - albeit it doesn't sound cool. I think its because it minimises false positives and robust. The study I was involved with (malaria) used k-means as the singular proof :-O  
  > **username_3**: In this study, the network phylogeny defined the groupings (it couldn't assign significance to it) and these groupings were explored via K-means. Trees (network trees) were there just because I like them, all the tree was saying that was statically significant was "these things cross-react" (hardly revolutionary). Otherwise it is a bit hard to describe k-means within a comments section, but let me know if you get the gist.  
  > **username_3**: I am doing supervised learning on quite similar data to yours at present but for viruses. However, this is a bit too close my current work to be providing detailed technical advice.  
  
  ---
  
  ### Answer by username_3 on 2019-04-17. Score: 1
  
  I've got it.
  The question you are asking is homogeneity between you target group and your general population of patients. What you want to show is "complete" homogeneity between the two groups regardless of any sub-sampling, ie. if there is groups within groups that might break your hypothesis.
  K-means is perfect here. The issue with K-means is designating the number of groups, but here thats irrelevant because its not the question you are asking. You are asking homogeneity and if your hypothesis is true no matter how many groups you designate for K-means, e.g. 2,3,4,5,6,7,8,9,10 (or maybe a more reduced sampling) homogeneity between target (treatment?) and general population is observed. This would therefore be a definitive analysing ASSUMING that you don't encounter a group where there is exclusively one population and the other is absent. Obviously the number of K-mean groups can't exceed the number of treated/target patients.
  
  It would be a really good application of k-means because it avoids the question,
  So why was that a priori selected?
  
  Personally I would first present your preferred a priori group size and then say same result for all group sizes. Its simple, recognised and neat and easy for a reviewer to follow. There are ML solutions to K-means, but I think "keep it simple".
  
  > **Comments:**
  > **username_0**: This makes allot of sense, and just knowing to google homogeneity has led me down what I think is the correct path, I am now struggling to find R packages that let me do this with explanations on how to sue them. I think clValid will work  
  > **username_3**: There is a direct function called 'kmeans' in R, but its not an approach I'd use unless I was reiterating within a data pipeline (R is great for automation). For this a general stats package SPSS or Stata is easier. ML k-means are in Sci-kit learn (Python), but this is really complicated and lots of things can go wrong. Glad this helped.  
  
  ---
  
  