Answers.Body: <p>What I'm not clear about is the definition of <code>y</code> - thats crucial to understanding what the code does. Just to note, although you've called the dataframe <code>data</code>, I've called it <code>df</code> (traditional shorthand). In terms of speed there will not be much in it between vectorised <code>pandas</code> and <code>numpy</code>. Explicitly using <code>iterrows</code> or else <code>apply</code> is when it slows down a lot.</p>
  <p>Possibilities of the definition of <code>y</code>.</p>
  <p>1.</p>
  <pre><code>y = df.drop(['Model', 'foldchange'], axis=1)
  </code></pre>
  <p>I think the above is more likely, or ...</p>
  <p>2.</p>
  <pre><code>y = df['fold change']
  </code></pre>
  <p>What I think is occurring is <code>np.mean</code> is performing a boolean operation on the feature columns. What I didn't understand though was this ...</p>
  <pre><code>abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
  </code></pre>
  <p>.. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because <code>y[~x]</code> will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within <code>pandas</code> rather than coercing <code>np</code> into a boolean operation. This is</p>
  <pre><code>import pandas as pd
  
  df = pd.read_csv(r'/pathtofile/megha.csv', delimiter=&quot;\t&quot;)
  
  feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
  df=df.drop(['foldchange'],axis=1)
  feature_level_partitions = ['L', 'G']
  filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
  df2 = df['Model'].to_frame().join(filtered)
  df2['mean'] = df2[feature].mean(axis=1)
  print(df2)
  </code></pre>
  <p>Here ~x would simply <code>1 - mean</code> as a new pandas column, i.e. <code>df2['~mean'] = df2['mean'] - 1</code></p>
  <p>The output of that code is:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>-</th>
  <th>Model</th>
  <th>A1BG</th>
  <th>A1CF</th>
  <th>A2M</th>
  <th>A2ML1</th>
  <th>A3GALT2</th>
  <th>A4GALT</th>
  <th>mean.</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>0</td>
  <td>IM00499</td>
  <td>0</td>
  <td>1.</td>
  <td>1.</td>
  <td>1.</td>
  <td>0.</td>
  <td>0.</td>
  <td>0.500000</td>
  </tr>
  <tr>
  <td>1</td>
  <td>IM00499.1</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0.333333</td>
  </tr>
  <tr>
  <td>2</td>
  <td>IM00980</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>3</td>
  <td>IM01529</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>4</td>
  <td>IM01624</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>5</td>
  <td>IM01240</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>6</td>
  <td>IM00807</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>7</td>
  <td>IM00999</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>8</td>
  <td>IM00983</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>9</td>
  <td>IM01076</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>10</td>
  <td>IM00803</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>11</td>
  <td>IM00125</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0.000000</td>
  </tr>
  <tr>
  <td>12</td>
  <td>IM00998</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0.166667</td>
  </tr>
  <tr>
  <td>13</td>
  <td>IM00997</td>
  <td>0</td>
  <td>1</td>
  <td>1</td>
  <td>1</td>
  <td>0</td>
  <td>1</td>
  <td>0.666667</td>
  </tr>
  <tr>
  <td>14</td>
  <td>IM00996</td>
  <td>0</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  <td>0</td>
  <td>1</td>
  <td>0.333333</td>
  </tr>
  </tbody>
  </table>
  </div><hr />
  <p>If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the <code>numpy</code> into explicit <code>pandas</code> operations, what is happening is there's a mix of <code>pandas</code> and <code>numpy</code> and I would recommend going down one route or the other.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T18:05:16.127Z
Answers.LastActivityDate: 2023-02-18T18:16:48.997Z
Answers.LastEditDate: 2023-02-18T18:16:48.997Z
Answers.UserDisplayName: M__
Body: <p>My data = <code>data</code></p>
  <pre><code>Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
  IM00499 N   L   L   G   N   N   0.285744
  IM00499.1   L   N   D   G   D   D   0.005046
  IM00980 N   N   N   N   N   N   -0.068949
  IM01529 N   N   L   N   N   N   0.065914
  IM01624 A   N   L   N   N   N   0.302100
  IM01240 N   D   N   D   N   G   -0.428324
  IM00807 N   L   A   N   D   N   0.257222
  IM00999 N   N   N   N   N   D   0.056268
  IM00983 N   N   N   N   N   N   0.107456
  IM01076 N   N   D   A   N   N   0.218301
  IM00803 N   D   N   N   N   D   -0.479368
  IM00125 N   N   A   N   N   A   0.270428
  IM00998 D   N   N   N   G   N   0.142748
  IM00997 N   L   G   G   N   G   -0.154540
  IM00996 N   L   N   N   N   L   -1.302023
  </code></pre>
  <p>The above data frame is divided into 5 categories = A, N, L, G and D</p>
  <p><code>feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']</code></p>
  <p>Feature_level_partitions are arrays of combination of classification values</p>
  <p><code>feature_level_partitions = ['A', 'N' , 'L',  'G',  'D'], ['D'], ['L', 'G']</code></p>
  <p>I was using the below <code>list comprehension</code> to calculate <code>Hedge's G</code>.</p>
  <pre><code>        hedges_g = np.array(
              [
                  Hedges_g(data[feature].isin(feature_level_partition), y)
                  for feature_level_partition in feature_level_partitions
              ],
              dtype=np.float64,
          )
  </code></pre>
  <p><code>Hedges_g</code> is defined as follows:</p>
  <pre><code>def Hedges_g(x, y):
      difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
      s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
      return difference_of_means / s_pooled_weighted
  </code></pre>
  <p>I am interested in knowing how many models there are within each category. So for example if I divide up the data into [&quot;D&quot;,&quot;L&quot;] on one side, and the rest on the other, I want to know how many models have either &quot;D&quot; OR &quot;L&quot; and how many models have any of the other categories and then don't calculate <code>Hegde's G</code> if the number on either side is less than 2.</p>
  <p>To answer the above query I tried the following code</p>
  <pre><code>[data[feature].isin(feature_level_partition).sum()
      for feature_level_partition in feature_level_partitions]
  </code></pre>
  <p>The above code gave me the number of models with the <code>feature_level_partitions</code>
  However I also want to know the number of models without the <code>feature_level_partitions</code></p>
  <p>I also want to create a python function which calculates both above numbers in one line and fit this code as a <code>if statement</code> in my above <code>list comprehension</code> function</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-19T19:35:35.313Z
Comments.Text: What you might think about @Megha is defining a new question in the context of both the comprehension and the models by defining a criteria for including 'foldchange' parameter.
Comments.UserDisplayName: M__
Tags: <python><pandas>
Title: Python list comprehension to calculate the set of categories for each row
ThreadText: # Python list comprehension to calculate the set of categories for each row
  
  (Asked by: username_0 on 2023-02-14)
  
  My data = `data`
  
  ```
  Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
  IM00499 N   L   L   G   N   N   0.285744
  IM00499.1   L   N   D   G   D   D   0.005046
  IM00980 N   N   N   N   N   N   -0.068949
  IM01529 N   N   L   N   N   N   0.065914
  IM01624 A   N   L   N   N   N   0.302100
  IM01240 N   D   N   D   N   G   -0.428324
  IM00807 N   L   A   N   D   N   0.257222
  IM00999 N   N   N   N   N   D   0.056268
  IM00983 N   N   N   N   N   N   0.107456
  IM01076 N   N   D   A   N   N   0.218301
  IM00803 N   D   N   N   N   D   -0.479368
  IM00125 N   N   A   N   N   A   0.270428
  IM00998 D   N   N   N   G   N   0.142748
  IM00997 N   L   G   G   N   G   -0.154540
  IM00996 N   L   N   N   N   L   -1.302023
  
  ```
  
  The above data frame is divided into 5 categories = A, N, L, G and D
  
  `feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']`
  
  Feature\_level\_partitions are arrays of combination of classification values
  
  `feature_level_partitions = ['A', 'N' , 'L', 'G', 'D'], ['D'], ['L', 'G']`
  
  I was using the below `list comprehension` to calculate `Hedge's G`.
  
  ```
          hedges_g = np.array(
              [
                  Hedges_g(data[feature].isin(feature_level_partition), y)
                  for feature_level_partition in feature_level_partitions
              ],
              dtype=np.float64,
          )
  
  ```
  
  `Hedges_g` is defined as follows:
  
  ```
  def Hedges_g(x, y):
      difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
      s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
      return difference_of_means / s_pooled_weighted
  
  ```
  
  I am interested in knowing how many models there are within each category. So for example if I divide up the data into ["D","L"] on one side, and the rest on the other, I want to know how many models have either "D" OR "L" and how many models have any of the other categories and then don't calculate `Hegde's G` if the number on either side is less than 2.
  
  To answer the above query I tried the following code
  
  ```
  [data[feature].isin(feature_level_partition).sum()
      for feature_level_partition in feature_level_partitions]
  
  ```
  
  The above code gave me the number of models with the `feature_level_partitions`
  However I also want to know the number of models without the `feature_level_partitions`
  
  I also want to create a python function which calculates both above numbers in one line and fit this code as a `if statement` in my above `list comprehension` function
  
  **Tags:** python, pandas
  
  > **Comments:**
  > **username_1**: What you might think about @username_0 is defining a new question in the context of both the comprehension and the models by defining a criteria for including 'foldchange' parameter.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-18. Score: 1 | Accepted answer
  
  What I'm not clear about is the definition of `y` - thats crucial to understanding what the code does. Just to note, although you've called the dataframe `data`, I've called it `df` (traditional shorthand). In terms of speed there will not be much in it between vectorised `pandas` and `numpy`. Explicitly using `iterrows` or else `apply` is when it slows down a lot.
  
  Possibilities of the definition of `y`.
  
  1.
  
  ```
  y = df.drop(['Model', 'foldchange'], axis=1)
  
  ```
  
  I think the above is more likely, or ...
  
  2.
  
  ```
  y = df['fold change']
  
  ```
  
  What I think is occurring is `np.mean` is performing a boolean operation on the feature columns. What I didn't understand though was this ...
  
  ```
  abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
  
  ```
  
  .. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because `y[~x]` will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within `pandas` rather than coercing `np` into a boolean operation. This is
  
  ```
  import pandas as pd
  
  df = pd.read_csv(r'/pathtofile/username_0.csv', delimiter="\t")
  
  feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
  df=df.drop(['foldchange'],axis=1)
  feature_level_partitions = ['L', 'G']
  filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
  df2 = df['Model'].to_frame().join(filtered)
  df2['mean'] = df2[feature].mean(axis=1)
  print(df2)
  
  ```
  
  Here ~x would simply `1 - mean` as a new pandas column, i.e. `df2['~mean'] = df2['mean'] - 1`
  
  The output of that code is:
  
  | - | Model | A1BG | A1CF | A2M | A2ML1 | A3GALT2 | A4GALT | mean. |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
  | 0 | IM00499 | 0 | 1. | 1. | 1. | 0. | 0. | 0.500000 |
  | 1 | IM00499.1 | 1 | 0 | 0 | 1 | 0 | 0 | 0.333333 |
  | 2 | IM00980 | 0 | 0 | 0 | 0 | 0 | 0 | 0.000000 |
  | 3 | IM01529 | 0 | 0 | 1 | 0 | 0 | 0 | 0.166667 |
  | 4 | IM01624 | 0 | 0 | 1 | 0 | 0 | 0 | 0.166667 |
  | 5 | IM01240 | 0 | 0 | 0 | 0 | 0 | 1 | 0.166667 |
  | 6 | IM00807 | 0 | 1 | 0 | 0 | 0 | 0 | 0.166667 |
  | 7 | IM00999 | 0 | 0 | 0 | 0 | 0 | 0 | 0.000000 |
  | 8 | IM00983 | 0 | 0 | 0 | 0 | 0 | 0 | 0.000000 |
  | 9 | IM01076 | 0 | 0 | 0 | 0 | 0 | 0 | 0.000000 |
  | 10 | IM00803 | 0 | 0 | 0 | 0 | 0 | 0 | 0.000000 |
  | 11 | IM00125 | 0 | 0 | 0 | 0 | 0 | 0 | 0.000000 |
  | 12 | IM00998 | 0 | 0 | 0 | 0 | 1 | 0 | 0.166667 |
  | 13 | IM00997 | 0 | 1 | 1 | 1 | 0 | 1 | 0.666667 |
  | 14 | IM00996 | 0 | 1 | 0 | 0 | 0 | 1 | 0.333333 |
  
  ---
  
  If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the `numpy` into explicit `pandas` operations, what is happening is there's a mix of `pandas` and `numpy` and I would recommend going down one route or the other.
  
  
  ---
  
  