Answers.Body: <p>This comprehensive analytical outline for the stats and all the stats questions could be captured inside 3 analyses. Implementing this will obviously take a bit of time, but there should not be any hidden caveats (transformations etc ...).</p>
  <p><strong>Question 3</strong></p>
  <p>The statistical test you want is K-means clustering based on the correlation score where the number of predefined groups is 3. The labels ... you will need to combine the group with the</p>
  <pre><code>Label          Value
  Group1-56-56   1
  ... 
  </code></pre>
  <p>This is an absolute solution to the data and really cool use of K-means.</p>
  <p><strong>Question 3c</strong>
  An easy solution is:</p>
  <ul>
  <li>perform a new K-means with e.g. 10 groups and find the group with highest correlation values.</li>
  </ul>
  <p>A better solution is:</p>
  <ul>
  <li>hierarchical clustering would identify specific groups and for this sub-question is a good analysis. The input is the correlation matrix, the output is a tree. HOWEVER, interpreting this could be difficult if you're not trained in reading trees.</li>
  </ul>
  <hr />
  <p><strong>Question 2</strong></p>
  <p>A 2-way ANOVA would be a really good solution to this data for Q2. The test would be the columns were DM1 DM2 and the rows Group1, Group2, Group3. Interpreting it would be fun because you'd also have to grapple with a concept called &quot;interaction&quot; - which could be important here. However, the ability to address a whole raft of questions within <em>one</em> analysis is what 2-way ANOVA would permit. There is a slight issue of unequal group sizes 1, 2 and 3, because it would be better if Group 1,2 and 3 etc... were equal in size. However, the raw data looks to be very amenable to ANOVA. You must not use the correlation index in this test - just the raw values.</p>
  <hr />
  <p>K-means clustering is in <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.vq.kmeans.html" rel="nofollow noreferrer">scipy</a> and <code>scikit-learn</code></p>
  <p>Hierarchical clustering is also in <a href="https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html" rel="nofollow noreferrer">scipy</a></p>
  <p>For ANOVA</p>
  <pre><code>import statsmodels.api as sm
  from statsmodels.formula.api import ols
  </code></pre>
  <p>There will be easy approaches to 2-way ANOVA e.g. SPSS and I think Excel (needs a plug in). I suspect that Hierarchical clustering will also be in a nice point and click package like SPSS (I don't use them, so I don't precisely know.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T16:49:20.003Z
Answers.Text: Is it possible for you to explain this in terms of code or any example code
Answers.UserDisplayName: Megha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T17:07:56.570Z
Answers.Text: Hi @Megha, you have asked an entire statistical project within a single question, so I answered as best I could.What I propose is to investigate the facets (K-means is a good start) and have a go and ask a specific question if you're stuck. I've updated the post with the location of the specific packages. You don't need to stick with Python, point and click packages are perfectly fine because this is end analysis. Beyond this there is no further process in a data pipeline
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-16T22:39:02.247Z
Answers.LastActivityDate: 2023-01-17T17:13:20.150Z
Answers.LastEditDate: 2023-01-17T17:13:20.150Z
Answers.UserDisplayName: M__
Body: <p>Subset of <code>df</code> before <code>groupby</code> looks like</p>
  <pre><code>Diabetes_Group  DM-1    DM-2    corr
  Group1  56  56  1
  Group1  56  58  0.611
  Group1  56  61  0.448
  Group1  56  63  0.3587
  Group1  58  56  0.611
  Group1  58  58  1
  Group1  58  61  0.60113
  Group1  58  63  0.534858
  Group1  61  56  0.448
  Group1  61  58  0.60113
  Group1  61  61  1
  Group1  61  63  0.622206
  Group1  63  56  0.3587
  Group1  63  58  0.534858
  Group1  63  61  0.622206
  Group1  63  63  1
  Group2  78  78  1
  Group2  78  85  0.622703
  Group2  78  94  0.622244
  Group2  78  23  0.687826
  Group2  85  78  0.622703
  Group2  85  85  1
  Group2  85  94  0.697313
  Group2  85  23  0.612089
  Group2  94  78  0.622244
  Group2  94  85  0.697313
  Group2  94  94  1
  Group2  94  23  0.603834
  Group2  23  78  0.687826
  Group2  23  85  0.612089
  Group2  23  94  0.603834
  Group2  23  23  1
  Group3  27  27  1
  Group3  27  80  0.118955
  Group3  27  147 0.32038
  Group3  27  9   0.335264
  Group3  80  27  0.118955
  Group3  80  80  1
  Group3  80  147 0.430287
  Group3  80  9   0.406426
  Group3  147 27  0.32038
  Group3  147 80  0.430287
  Group3  147 147 1
  Group3  147 9   0.546452
  Group3  9   27  0.335264
  Group3  9   80  0.406426
  Group3  9   147 0.546452
  Group3  9   9   1
  </code></pre>
  <ol>
  <li><p>What statistical test can be used to do comparative analysis between the different ID's (<code>DM1 or DM2</code>) to check which ID pair has weaker/stronger correlation within each diabetes_group?</p>
  </li>
  <li><p>Is there any statistical test to categorise the data into three categories:</p>
  <p>(a) <code>pairs of DM1 AND DM2</code> which are highly correlated than others.</p>
  <p>(b) pairs that are not highly correlated</p>
  <p>(c) For the category 1 pairs (highly correlated), is there any particular ID that outperforms than other ID's among different groups?</p>
  </li>
  </ol>
  
Tags: <python><r>
Title: How to statistically test the strength and weakness of a correlated pair?
ThreadText: # How to statistically test the strength and weakness of a correlated pair?
  
  (Asked by: username_0 on 2023-01-16)
  
  Subset of `df` before `groupby` looks like
  
  ```
  Diabetes_Group  DM-1    DM-2    corr
  Group1  56  56  1
  Group1  56  58  0.611
  Group1  56  61  0.448
  Group1  56  63  0.3587
  Group1  58  56  0.611
  Group1  58  58  1
  Group1  58  61  0.60113
  Group1  58  63  0.534858
  Group1  61  56  0.448
  Group1  61  58  0.60113
  Group1  61  61  1
  Group1  61  63  0.622206
  Group1  63  56  0.3587
  Group1  63  58  0.534858
  Group1  63  61  0.622206
  Group1  63  63  1
  Group2  78  78  1
  Group2  78  85  0.622703
  Group2  78  94  0.622244
  Group2  78  23  0.687826
  Group2  85  78  0.622703
  Group2  85  85  1
  Group2  85  94  0.697313
  Group2  85  23  0.612089
  Group2  94  78  0.622244
  Group2  94  85  0.697313
  Group2  94  94  1
  Group2  94  23  0.603834
  Group2  23  78  0.687826
  Group2  23  85  0.612089
  Group2  23  94  0.603834
  Group2  23  23  1
  Group3  27  27  1
  Group3  27  80  0.118955
  Group3  27  147 0.32038
  Group3  27  9   0.335264
  Group3  80  27  0.118955
  Group3  80  80  1
  Group3  80  147 0.430287
  Group3  80  9   0.406426
  Group3  147 27  0.32038
  Group3  147 80  0.430287
  Group3  147 147 1
  Group3  147 9   0.546452
  Group3  9   27  0.335264
  Group3  9   80  0.406426
  Group3  9   147 0.546452
  Group3  9   9   1
  
  ```
  
  1. What statistical test can be used to do comparative analysis between the different ID's (`DM1 or DM2`) to check which ID pair has weaker/stronger correlation within each diabetes\_group?
  2. Is there any statistical test to categorise the data into three categories:
  
     (a) `pairs of DM1 AND DM2` which are highly correlated than others.
  
     (b) pairs that are not highly correlated
  
     (c) For the category 1 pairs (highly correlated), is there any particular ID that outperforms than other ID's among different groups?
  
  **Tags:** python, r
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-16. Score: 2 | Accepted answer
  
  This comprehensive analytical outline for the stats and all the stats questions could be captured inside 3 analyses. Implementing this will obviously take a bit of time, but there should not be any hidden caveats (transformations etc ...).
  
  **Question 3**
  
  The statistical test you want is K-means clustering based on the correlation score where the number of predefined groups is 3. The labels ... you will need to combine the group with the
  
  ```
  Label          Value
  Group1-56-56   1
  ... 
  
  ```
  
  This is an absolute solution to the data and really cool use of K-means.
  
  **Question 3c**
  An easy solution is:
  
  * perform a new K-means with e.g. 10 groups and find the group with highest correlation values.
  
  A better solution is:
  
  * hierarchical clustering would identify specific groups and for this sub-question is a good analysis. The input is the correlation matrix, the output is a tree. HOWEVER, interpreting this could be difficult if you're not trained in reading trees.
  
  ---
  
  **Question 2**
  
  A 2-way ANOVA would be a really good solution to this data for Q2. The test would be the columns were DM1 DM2 and the rows Group1, Group2, Group3. Interpreting it would be fun because you'd also have to grapple with a concept called "interaction" - which could be important here. However, the ability to address a whole raft of questions within *one* analysis is what 2-way ANOVA would permit. There is a slight issue of unequal group sizes 1, 2 and 3, because it would be better if Group 1,2 and 3 etc... were equal in size. However, the raw data looks to be very amenable to ANOVA. You must not use the correlation index in this test - just the raw values.
  
  ---
  
  K-means clustering is in [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.vq.kmeans.html) and `scikit-learn`
  
  Hierarchical clustering is also in [scipy](https://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html)
  
  For ANOVA
  
  ```
  import statsmodels.api as sm
  from statsmodels.formula.api import ols
  
  ```
  
  There will be easy approaches to 2-way ANOVA e.g. SPSS and I think Excel (needs a plug in). I suspect that Hierarchical clustering will also be in a nice point and click package like SPSS (I don't use them, so I don't precisely know.)
  
  > **Comments:**
  > **username_0**: Is it possible for you to explain this in terms of code or any example code  
  > **username_1**: Hi @username_0, you have asked an entire statistical project within a single question, so I answered as best I could.What I propose is to investigate the facets (K-means is a good start) and have a go and ask a specific question if you're stuck. I've updated the post with the location of the specific packages. You don't need to stick with Python, point and click packages are perfectly fine because this is end analysis. Beyond this there is no further process in a data pipeline  
  
  ---
  
  