Answers.Body: <p>Reformatting the data for the KW and doing the test is easy.</p>
  <p>Using &quot;guide_1&quot; alone and pivot() to get A = [1, 0.426115 ...</p>
  <pre><code>df2 = df.pivot(columns='guide_1', values='Correlation')
  </code></pre>
  <p>This will now present A, B, C, D etc ... into columns...</p>
  <pre><code>x = df2[['A']].dropna() 
  y = df2[['B']].dropna() 
  z = df2[['C']].dropna() 
  a = df2[['D']].dropna() 
  
  stats.kruskal(x, y, z, a)
  </code></pre>
  <p>That will work ... whether that is a good test is a complicated question because this is log transformed data. Transformation is used in parametric tests. Removing the log transformation is strongly recommended because KW works with raw data.</p>
  <p>You can switch the calculation to <code>guide_1_new</code> instead. It will give a different answer ... but I don't know the experiment. The original guide looks cool.</p>
  <p>Looking at the data - the test should work because I believe KW is 'unpaired' (you need to check) and looking at the value the distribution looks pretty similar between A, B, C and D. However, just because the KW says 'yes' doesn't mean thats a robust answer (log data). If the untransformed data says 'yes' ... thats a robust answer for KW. Just check the 'unpaired' stuff - thats important for the result.</p>
  <hr />
  <p>Personal note, I originally used a third df but was not needed,</p>
  <pre><code>df3 = pd.concat([df2[col].sort_values().reset_index(drop=True) for col in df2], axis=1, ignore_index=True)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-19T11:37:33.630Z
Answers.Text: I'm a little confused here, it looks to me in the OP like the numeric input is correlation coefficients from comparing A to A, A to B, etc. So for A to A, you get an unsurprising correlation of 1, which is now included in the A group, which doesn't seem like a very informative input. Does OP really care about whether `guide_1` group A is more correlated to other groups than those groups are among themselves (which is what this is testing for, effectively)? Doesn't make sense to me, shouldn't OP go back to raw (pre-correlation) data? Probably missing something here.
Answers.UserDisplayName: Maximilian Press
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-19T12:37:21.490Z
Answers.Text: @MaximilianPress KW will give a result representing the data. Agreed non-parametrics prefer raw data. The presence of A->A , B->B will of course bias the test, but the gist is A ~ B ~ C ~ D - and will not be 'thrown out of court'. Capturing log-change or non-log change isn't easy. A 2-way ANOVA type (4 columns vs 2 rows) would do this - absolutely, but its complicated and non-parametrics are not good at this calculation (Friedman test, statisticians avoid it). Using 1-way ANOVA on this specific data would be a mistake. Getting it 'spot on' is an investment. Right now 'it's ok'.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-18T22:56:15.793Z
Answers.LastActivityDate: 2023-01-19T02:08:56.290Z
Answers.LastEditDate: 2023-01-19T02:08:56.290Z
Answers.UserDisplayName: M__
Body: <p><code>df</code></p>
  <pre><code>gene_name   guide_1 guide_1_new Correlation
  MMP-1   A   A   1
  MMP-1   A   B   0.426115
  MMP-1   A   C   0.522499
  MMP-1   A   D   0.431587
  MMP-1   B   A   0.426115
  MMP-1   B   B   1
  MMP-1   B   C   0.60113
  MMP-1   B   D   0.534858
  MMP-1   C   A   0.522499
  MMP-1   C   B   0.60113
  MMP-1   C   C   1
  MMP-1   C   D   0.622206
  MMP-1   D   A   0.431587
  MMP-1   D   B   0.534858
  MMP-1   D   C   0.622206
  MMP-1   D   D   1
  </code></pre>
  <p>I used four guides in an experiment (data = <code>df</code>). I calculated correlation between them based on the their log fold changes. I want to check which guide is outperforming or underperforming in comparison to other three guides in my experiment. I tried Kruskal and Wallis test to do this analysis with the script given below. However, I got an error <code>ValueError: Samples must be one-dimensional</code>. Also is this the right test to do this analysis?</p>
  <p><code>My script</code></p>
  <pre><code>from scipy import stats
  
  x = df[['guide_1']]
  y = df[['guide_1_new']]
  z = df[['Correlation']]
  stats.kruskal(x, y, z)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-18T21:32:47.053Z
Comments.Text: I am a little confused, you are passing the columns of `guide_1` and `guide_1_new` as samples to `scipy.stats.kruskal`? Those are non-numeric data. And the correlations, while numeric, suggest to me that you are doing something a little different from what Kruskal-Wallis is implementing. It looks like you might be trying to pass in the categories as vectors of strings, but I don't think it works that way (?). And I don't think applying it to correlation coefficients will be very interpretable. Not an expert on `scipy.stats` but this doesn't look right. I'd suggest going back to raw data.
Comments.UserDisplayName: Maximilian Press
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-18T23:18:51.463Z
Comments.Text: Answered below, if you are stuck I can talk you through the calculation if you perform a df.to_dict() and post the output.
Comments.UserDisplayName: M__
Tags: <r><pandas>
Title: How to fix the Error in Kruskal Wallis rank sum test for comparing the guides in python? Also to check whether using the right test for my data?
ThreadText: # How to fix the Error in Kruskal Wallis rank sum test for comparing the guides in python? Also to check whether using the right test for my data?
  
  (Asked by: username_0 on 2023-01-18)
  
  `df`
  
  ```
  gene_name   guide_1 guide_1_new Correlation
  MMP-1   A   A   1
  MMP-1   A   B   0.426115
  MMP-1   A   C   0.522499
  MMP-1   A   D   0.431587
  MMP-1   B   A   0.426115
  MMP-1   B   B   1
  MMP-1   B   C   0.60113
  MMP-1   B   D   0.534858
  MMP-1   C   A   0.522499
  MMP-1   C   B   0.60113
  MMP-1   C   C   1
  MMP-1   C   D   0.622206
  MMP-1   D   A   0.431587
  MMP-1   D   B   0.534858
  MMP-1   D   C   0.622206
  MMP-1   D   D   1
  
  ```
  
  I used four guides in an experiment (data = `df`). I calculated correlation between them based on the their log fold changes. I want to check which guide is outperforming or underperforming in comparison to other three guides in my experiment. I tried Kruskal and Wallis test to do this analysis with the script given below. However, I got an error `ValueError: Samples must be one-dimensional`. Also is this the right test to do this analysis?
  
  `My script`
  
  ```
  from scipy import stats
  
  x = df[['guide_1']]
  y = df[['guide_1_new']]
  z = df[['Correlation']]
  stats.kruskal(x, y, z)
  
  ```
  
  **Tags:** r, pandas
  
  > **Comments:**
  > **username_1**: I am a little confused, you are passing the columns of `guide_1` and `guide_1_new` as samples to `scipy.stats.kruskal`? Those are non-numeric data. And the correlations, while numeric, suggest to me that you are doing something a little different from what Kruskal-Wallis is implementing. It looks like you might be trying to pass in the categories as vectors of strings, but I don't think it works that way (?). And I don't think applying it to correlation coefficients will be very interpretable. Not an expert on `scipy.stats` but this doesn't look right. I'd suggest going back to raw data.  
  > **username_2**: Answered below, if you are stuck I can talk you through the calculation if you perform a df.to_dict() and post the output.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-01-18. Score: 3 | Accepted answer
  
  Reformatting the data for the KW and doing the test is easy.
  
  Using "guide\_1" alone and pivot() to get A = [1, 0.426115 ...
  
  ```
  df2 = df.pivot(columns='guide_1', values='Correlation')
  
  ```
  
  This will now present A, B, C, D etc ... into columns...
  
  ```
  x = df2[['A']].dropna() 
  y = df2[['B']].dropna() 
  z = df2[['C']].dropna() 
  a = df2[['D']].dropna() 
  
  stats.kruskal(x, y, z, a)
  
  ```
  
  That will work ... whether that is a good test is a complicated question because this is log transformed data. Transformation is used in parametric tests. Removing the log transformation is strongly recommended because KW works with raw data.
  
  You can switch the calculation to `guide_1_new` instead. It will give a different answer ... but I don't know the experiment. The original guide looks cool.
  
  Looking at the data - the test should work because I believe KW is 'unpaired' (you need to check) and looking at the value the distribution looks pretty similar between A, B, C and D. However, just because the KW says 'yes' doesn't mean thats a robust answer (log data). If the untransformed data says 'yes' ... thats a robust answer for KW. Just check the 'unpaired' stuff - thats important for the result.
  
  ---
  
  Personal note, I originally used a third df but was not needed,
  
  ```
  df3 = pd.concat([df2[col].sort_values().reset_index(drop=True) for col in df2], axis=1, ignore_index=True)
  
  ```
  
  > **Comments:**
  > **username_1**: I'm a little confused here, it looks to me in the OP like the numeric input is correlation coefficients from comparing A to A, A to B, etc. So for A to A, you get an unsurprising correlation of 1, which is now included in the A group, which doesn't seem like a very informative input. Does OP really care about whether `guide_1` group A is more correlated to other groups than those groups are among themselves (which is what this is testing for, effectively)? Doesn't make sense to me, shouldn't OP go back to raw (pre-correlation) data? Probably missing something here.  
  > **username_2**: @username_1 KW will give a result representing the data. Agreed non-parametrics prefer raw data. The presence of A->A , B->B will of course bias the test, but the gist is A ~ B ~ C ~ D - and will not be 'thrown out of court'. Capturing log-change or non-log change isn't easy. A 2-way ANOVA type (4 columns vs 2 rows) would do this - absolutely, but its complicated and non-parametrics are not good at this calculation (Friedman test, statisticians avoid it). Using 1-way ANOVA on this specific data would be a mistake. Getting it 'spot on' is an investment. Right now 'it's ok'.  
  
  ---
  
  