Answers.Body: <p>Alright, so there are a number of problematic patterns in your code - as far as I understand what you are trying to do. Next time, try to post a reproducible example that people can use and more people will be willing to help.</p>
  <h3>L18-21:</h3>
  <pre class="lang-py prettyprint-override"><code>combination_labels = []
  combination_counts = []
  
  for lineage in lineages:
  </code></pre>
  <p>Declaring these two lists <em>before</em> the loop, then appending all your combination labels inside the loop means that these two lists will contain all labels and counts for all runs (all your lineages). If I am understanding your intentions correctly this is probably not something you want. The simple fix here is to just move them into the for-loop.</p>
  <h3>L29-30:</h3>
  <pre class="lang-py prettyprint-override"><code>for mutation in combination:
      combination_df = combination_df[combination_df[mutation] == 1]
  </code></pre>
  <p>Here you are iteratively checking if the value for each mutation is set to '1', but you are also <em>overwriting</em> your <code>combination_df</code> variable at each iteration of the loop. After this loop, <code>combination_df</code> will be whatever mutation last was set to '1' in your current combination. I will get back to solving this in my solution below.</p>
  <h3>L37-44:</h3>
  <pre class="lang-py prettyprint-override"><code>combination_labels.append('_'.join(combination))
  combination_counts.append(combination_df.shape[0])
  
  out_df = pd.DataFrame({'combination':combination_labels,
                     'count':combination_counts})
  out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
  out_df = out_df.sort_values('percentage', ascending = False)   
  
  out_df.to_csv('Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv',
           header = True, 
           index = False)
  </code></pre>
  <p>Since this part is again scoped inside the for-loop, each iteration (each combination) will overwrite the same <code>'Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv'</code> file. This you also want to move out of the <code>for combination in ...</code> loop.</p>
  <hr />
  <p>A big concern is also performance. You have 2000-something combinations and 100,000 lines in your file. As you loop over combinations, you also process the DataFrame each time, when this is a proble you can solve in one iteration. It's not much data so it's still fine, but these habits are still good to develop early.</p>
  <p>Now here's an idea for a rewrite that should work better. I don't have a csv from you to test with, so there might (proabably are) still things to fix, but hopefully it'll give you a starting point:</p>
  <pre class="lang-py prettyprint-override"><code>mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
               'S477', 'S477N', 'T478I', 'V483A']
  
  combinations = []
  for M in range(1, len(mutations)+1):
      for subset in itertools.combinations(mutations, M):
          combinations.append(subset)  
          
  combinations = ['_'.join(sorted(x)) for x in combinations]
  
  root = &quot;C:&quot;
  
  os.chdir(root)
  
  lineages = os.listdir('Results')
  
  for lineage in lineages:
  
      out_df = pd.DataFrame({&quot;Count&quot;: 0}, index=combinations)
      
      df = pd.read_csv('Results/' + lineage, index_col=&quot;lineage&quot;).dropna()
      
      for index, row in df.iterrows():
          combination = '_'.join(sorted(df.columns[row==1]))
          out_df.loc[combination, &quot;Count&quot;] += 1
  
      out_df['Percentage'] = (out_df['Count'] / df.shape[0]) * 100
      out_df = out_df.sort_values('Percentage', ascending = False)   
  
      out_df.to_csv('Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv', header = True, 
          index = True)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-24T22:15:39.237Z
Answers.Text: Absolutely amazingly detailed reply, thank you so much! It's worked exactly how I needed it! Not sure how to mark this as answered, as I'm very new to this forum - perhaps a mod could help here?
  Thank you again, sincerely!
Answers.UserDisplayName: Theo Jones
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-25T06:54:38.240Z
Answers.Text: [This](https://stackoverflow.com/help/someone-answers) might help you with that
Answers.UserDisplayName: Bastian Schiffthaler
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-24T14:42:32.840Z
Answers.LastActivityDate: 2020-10-24T14:54:55.073Z
Answers.LastEditDate: 2020-10-24T14:54:55.073Z
Answers.UserDisplayName: Bastian Schiffthaler
Body: <pre><code>mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
               'S477', 'S477N', 'T478I', 'V483A']
  
  combinations = []
  for M in range(1, len(mutations)+1):
      for subset in itertools.combinations(mutations, M):
          combinations.append(subset)
          
          combinations = ['_'.join(sorted(x)) for x in combinations]
          combinations = [x.split('_') for x in list(set(combinations))]
  
  root = &quot;C:&quot;
  
  os.chdir(root)
  
  lineages = os.listdir('Results')
  
  combination_labels = []
  combination_counts = []
  
  for lineage in lineages:
      
      df = pd.read_csv('Results/' + lineage).dropna()
      
      for combination in combinations:
      
              combination_df = df[list(df)]
          
              for mutation in combination:
                  combination_df = combination_df[combination_df[mutation] == 1]
      
          #print(combination_df.shape[0])
          
              combination_labels.append('_'.join(combination))
              combination_counts.append(combination_df.shape[0])
          
              out_df = pd.DataFrame({'combination':combination_labels,
                                 'count':combination_counts})
              out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
              out_df = out_df.sort_values('percentage', ascending = False)   
          
              out_df.to_csv('Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv',
                       header = True, 
                       index = False)
  </code></pre>
  <p>The input CSV</p>
  <pre><code>lineage,A222V,D614G,E484Q,E780Q,G476S,L18F,N439K,S477,S477N,T478I,V483A
  417941,0,1,0,0,0,0,0,0,0,0,0
  </code></pre>
  <p>Output CSV</p>
  <pre><code>combination,count,percentage
  D614G,87355,90.7084929856806
  </code></pre>
  <p>My above code is used to count all occurences of combinations of spike protein mutations</p>
  <ul>
  <li>to determine the genotypes.
  However, looking at the values processed into the .csv file, i can see ' D614G' is being counted even when
  it is in combination with other combinations.</li>
  </ul>
  <p>My question is how do i ensure once something has been counted (a row in the imported .csv file)
  it is not counted for a second time?</p>
  <p>OR possibly how can I edit this code to prevent counting of singular mutations even when presented within a combination?</p>
  <p>Thanks in advance</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-23T15:48:09.997Z
Comments.Text: Nice to see a traditional python question based on pandas. The top of your code is chopped off. Could you inset it please?
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-23T16:53:44.513Z
Comments.Text: Could you also add what a row in your CSV looks like?
Comments.UserDisplayName: Bastian Schiffthaler
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-23T17:01:31.223Z
Comments.Text: Input / output csv?
Comments.UserDisplayName: Theo Jones
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-23T23:33:37.420Z
Comments.Text: The input CSV in your Result folder is only two lines per file?
Comments.UserDisplayName: zorbax
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-24T10:35:19.720Z
Comments.Text: The input file has about 97,000 lines of data - each for a strain of SARSCOV2. The output file has around 54 lines, as genotypes but this should be quite a few less
Comments.UserDisplayName: Theo Jones
Tags: <python><genomics><covid-19><sars-cov-2><mutations>
Title: >My counter is counting genotypic combination occurences more than once, how do I ensure it counts one combination and doesnt count it again?
ThreadText: # >My counter is counting genotypic combination occurences more than once, how do I ensure it counts one combination and doesnt count it again?
  
  (Asked by: username_0 on 2020-10-23)
  
  ```
  mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
               'S477', 'S477N', 'T478I', 'V483A']
  
  combinations = []
  for M in range(1, len(mutations)+1):
      for subset in itertools.combinations(mutations, M):
          combinations.append(subset)
          
          combinations = ['_'.join(sorted(x)) for x in combinations]
          combinations = [x.split('_') for x in list(set(combinations))]
  
  root = "C:"
  
  os.chdir(root)
  
  lineages = os.listdir('Results')
  
  combination_labels = []
  combination_counts = []
  
  for lineage in lineages:
      
      df = pd.read_csv('Results/' + lineage).dropna()
      
      for combination in combinations:
      
              combination_df = df[list(df)]
          
              for mutation in combination:
                  combination_df = combination_df[combination_df[mutation] == 1]
      
          #print(combination_df.shape[0])
          
              combination_labels.append('_'.join(combination))
              combination_counts.append(combination_df.shape[0])
          
              out_df = pd.DataFrame({'combination':combination_labels,
                                 'count':combination_counts})
              out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
              out_df = out_df.sort_values('percentage', ascending = False)   
          
              out_df.to_csv('Results_2/' + lineage.replace(".csv", "") + '_3.csv',
                       header = True, 
                       index = False)
  
  ```
  
  The input CSV
  
  ```
  lineage,A222V,D614G,E484Q,E780Q,G476S,L18F,N439K,S477,S477N,T478I,V483A
  417941,0,1,0,0,0,0,0,0,0,0,0
  
  ```
  
  Output CSV
  
  ```
  combination,count,percentage
  D614G,87355,90.7084929856806
  
  ```
  
  My above code is used to count all occurences of combinations of spike protein mutations
  
  * to determine the genotypes.
    However, looking at the values processed into the .csv file, i can see ' D614G' is being counted even when
    it is in combination with other combinations.
  
  My question is how do i ensure once something has been counted (a row in the imported .csv file)
  it is not counted for a second time?
  
  OR possibly how can I edit this code to prevent counting of singular mutations even when presented within a combination?
  
  Thanks in advance
  
  **Tags:** python, genomics, covid-19, sars-cov-2, mutations
  
  > **Comments:**
  > **username_1**: Nice to see a traditional python question based on pandas. The top of your code is chopped off. Could you inset it please?  
  > **username_2**: Could you also add what a row in your CSV looks like?  
  > **username_0**: Input / output csv?  
  > **username_3**: The input CSV in your Result folder is only two lines per file?  
  > **username_0**: The input file has about 97,000 lines of data - each for a strain of SARSCOV2. The output file has around 54 lines, as genotypes but this should be quite a few less  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-24. Score: 3 | Accepted answer
  
  Alright, so there are a number of problematic patterns in your code - as far as I understand what you are trying to do. Next time, try to post a reproducible example that people can use and more people will be willing to help.
  
  ### L18-21:
  
  ```py
  combination_labels = []
  combination_counts = []
  
  for lineage in lineages:
  
  ```
  
  Declaring these two lists *before* the loop, then appending all your combination labels inside the loop means that these two lists will contain all labels and counts for all runs (all your lineages). If I am understanding your intentions correctly this is probably not something you want. The simple fix here is to just move them into the for-loop.
  
  ### L29-30:
  
  ```py
  for mutation in combination:
      combination_df = combination_df[combination_df[mutation] == 1]
  
  ```
  
  Here you are iteratively checking if the value for each mutation is set to '1', but you are also *overwriting* your `combination_df` variable at each iteration of the loop. After this loop, `combination_df` will be whatever mutation last was set to '1' in your current combination. I will get back to solving this in my solution below.
  
  ### L37-44:
  
  ```py
  combination_labels.append('_'.join(combination))
  combination_counts.append(combination_df.shape[0])
  
  out_df = pd.DataFrame({'combination':combination_labels,
                     'count':combination_counts})
  out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
  out_df = out_df.sort_values('percentage', ascending = False)   
  
  out_df.to_csv('Results_2/' + lineage.replace(".csv", "") + '_3.csv',
           header = True, 
           index = False)
  
  ```
  
  Since this part is again scoped inside the for-loop, each iteration (each combination) will overwrite the same `'Results_2/' + lineage.replace(".csv", "") + '_3.csv'` file. This you also want to move out of the `for combination in ...` loop.
  
  ---
  
  A big concern is also performance. You have 2000-something combinations and 100,000 lines in your file. As you loop over combinations, you also process the DataFrame each time, when this is a proble you can solve in one iteration. It's not much data so it's still fine, but these habits are still good to develop early.
  
  Now here's an idea for a rewrite that should work better. I don't have a csv from you to test with, so there might (proabably are) still things to fix, but hopefully it'll give you a starting point:
  
  ```py
  mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
               'S477', 'S477N', 'T478I', 'V483A']
  
  combinations = []
  for M in range(1, len(mutations)+1):
      for subset in itertools.combinations(mutations, M):
          combinations.append(subset)  
          
  combinations = ['_'.join(sorted(x)) for x in combinations]
  
  root = "C:"
  
  os.chdir(root)
  
  lineages = os.listdir('Results')
  
  for lineage in lineages:
  
      out_df = pd.DataFrame({"Count": 0}, index=combinations)
      
      df = pd.read_csv('Results/' + lineage, index_col="lineage").dropna()
      
      for index, row in df.iterrows():
          combination = '_'.join(sorted(df.columns[row==1]))
          out_df.loc[combination, "Count"] += 1
  
      out_df['Percentage'] = (out_df['Count'] / df.shape[0]) * 100
      out_df = out_df.sort_values('Percentage', ascending = False)   
  
      out_df.to_csv('Results_2/' + lineage.replace(".csv", "") + '_3.csv', header = True, 
          index = True)
  
  ```
  
  > **Comments:**
  > **username_0**: Absolutely amazingly detailed reply, thank you so much! It's worked exactly how I needed it! Not sure how to mark this as answered, as I'm very new to this forum - perhaps a mod could help here? Thank you again, sincerely!  
  > **username_2**: [This](https://stackoverflow.com/help/someone-answers) might help you with that  
  
  ---
  
  