Answers.Body: <p>Just to mention I personally think there is a higher loop above the loop stated above.</p>
  <p>I've not looked at each bit of code in detail but
  there's a bug is on line 4:</p>
  <pre><code>results = results.append({&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}, ignore_index=True)
  </code></pre>
  <p>Firstly, this is fashionable coding because you are dumping straight into a <code>pandas</code> dataframe (described below). The correction is simply ... declare <code>pandas</code> in advance</p>
  <pre><code>import pandas as pd
  results = pd.DataFrame()
  p_value = # code missing
  if p_value &lt; 0.05:
      for gene in list(pathway):
          results.append({&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}, ignore_index=True)
  </code></pre>
  <p>The thing is if there is a higher loop above <code>list(pathway)</code> then <code>results = results.append(....</code> will over-ride the last set of results. This would explain why results 'disappear' because if the last data in the loop is a blank - or very little output, all previous stuff gets deleted.</p>
  <p>I personally would use a dictionary here. Thus</p>
  <pre><code># declare the dictionary before all loops     
  results = {&quot;Gene&quot;: [], &quot;Model&quot;: [], &quot;Enriched Pathway&quot;: []} 
  p_value = # code missing
  if p_value &lt; 0.05:
      for gene in list(pathway):
         genedict = {&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}
         for k,v in mydicti.values():
             results[v].append(v)
  </code></pre>
  <p>... you could then transfer this to <code>pandas</code> once the loops are complete via,</p>
  <pre><code>df = pd.DataFrame(genedict)
  </code></pre>
  <p>I do suspect there's other issues, but as a general rule <code>results = results.append(...</code> is better declaring <code>results</code> first and then appending to <code>results</code>, unless you've a very specific reason (e.g. writing each loop into a separate file). The <code>pandas</code> thing is kinda clever for big data sets, because <code>pandas</code> is fast, whilst dictionaries are slow - for big data (like &gt;&gt;500 000 big). The interchangeability between <code>dictionaries</code> and <code>pandas</code> means it's kinda easier conceptually (and code-wise) to switch at the coders convenience in my personal opinion.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T23:31:20.090Z
Answers.LastActivityDate: 2023-04-05T23:42:48.990Z
Answers.LastEditDate: 2023-04-05T23:42:48.990Z
Answers.UserDisplayName: M__
Body: <p>I was doing <code>pathway enrichment analysis</code> using the below code</p>
  <pre><code>p_value = scipy.stats.hypergeom.sf(len(intersection) - 1, geneset_size, len(pathway), len(hits))
          if p_value &lt; 0.05:
              for gene in list(pathway):
                  results = results.append({&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}, ignore_index=True)
  </code></pre>
  <p>where,</p>
  <pre><code>hits = df[&quot;Gene&quot;][np.logical_and(df[&quot;Model&quot;] == model, df[&quot;p_value_anova&quot;] &lt; 0.05)]
  intersection = pathway.intersection(hits)
  Model = model name (like A, B, C) in input file
  Gene = Gene from the input gene
  Enriched pathway=Pathways that are enriched from the Pathway input file provided
  </code></pre>
  <p>Now the problem is that earlier this checking pathway enrichment on the basis of p_value in <code>hits</code> was working but now using cutoff (0.05) is resulting in loss of all pathways. I am looking to test the enrichment using the statistical test that will work on continous measure than binarizing the P-value.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-05T20:32:05.760Z
Comments.Text: Hi @Megha, do you have a small amount of sample data?
Comments.UserDisplayName: M__
Tags: <python><r><pathway><go-enrichment><pandas>
Title: How to get enriched pathways in the data using continous statistic measure?
ThreadText: # How to get enriched pathways in the data using continous statistic measure?
  
  (Asked by: username_0 on 2023-04-05)
  
  I was doing `pathway enrichment analysis` using the below code
  
  ```
  p_value = scipy.stats.hypergeom.sf(len(intersection) - 1, geneset_size, len(pathway), len(hits))
          if p_value < 0.05:
              for gene in list(pathway):
                  results = results.append({"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}, ignore_index=True)
  
  ```
  
  where,
  
  ```
  hits = df["Gene"][np.logical_and(df["Model"] == model, df["p_value_anova"] < 0.05)]
  intersection = pathway.intersection(hits)
  Model = model name (like A, B, C) in input file
  Gene = Gene from the input gene
  Enriched pathway=Pathways that are enriched from the Pathway input file provided
  
  ```
  
  Now the problem is that earlier this checking pathway enrichment on the basis of p\_value in `hits` was working but now using cutoff (0.05) is resulting in loss of all pathways. I am looking to test the enrichment using the statistical test that will work on continous measure than binarizing the P-value.
  
  **Tags:** python, r, pathway, go-enrichment, pandas
  
  > **Comments:**
  > **username_1**: Hi @username_0, do you have a small amount of sample data?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-05. Score: 1 | Accepted answer
  
  Just to mention I personally think there is a higher loop above the loop stated above.
  
  I've not looked at each bit of code in detail but
  there's a bug is on line 4:
  
  ```
  results = results.append({"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}, ignore_index=True)
  
  ```
  
  Firstly, this is fashionable coding because you are dumping straight into a `pandas` dataframe (described below). The correction is simply ... declare `pandas` in advance
  
  ```
  import pandas as pd
  results = pd.DataFrame()
  p_value = # code missing
  if p_value < 0.05:
      for gene in list(pathway):
          results.append({"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}, ignore_index=True)
  
  ```
  
  The thing is if there is a higher loop above `list(pathway)` then `results = results.append(....` will over-ride the last set of results. This would explain why results 'disappear' because if the last data in the loop is a blank - or very little output, all previous stuff gets deleted.
  
  I personally would use a dictionary here. Thus
  
  ```
  # declare the dictionary before all loops     
  results = {"Gene": [], "Model": [], "Enriched Pathway": []} 
  p_value = # code missing
  if p_value < 0.05:
      for gene in list(pathway):
         genedict = {"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}
         for k,v in mydicti.values():
             results[v].append(v)
  
  ```
  
  ... you could then transfer this to `pandas` once the loops are complete via,
  
  ```
  df = pd.DataFrame(genedict)
  
  ```
  
  I do suspect there's other issues, but as a general rule `results = results.append(...` is better declaring `results` first and then appending to `results`, unless you've a very specific reason (e.g. writing each loop into a separate file). The `pandas` thing is kinda clever for big data sets, because `pandas` is fast, whilst dictionaries are slow - for big data (like >>500 000 big). The interchangeability between `dictionaries` and `pandas` means it's kinda easier conceptually (and code-wise) to switch at the coders convenience in my personal opinion.
  
  
  ---
  
  