Answers.Body: <p>You don't need <code>g2</code> to go from <code>0</code> to <code>din.index</code>, just from <code>0</code> to <code>g1 - 1</code>. This way, you'll end up calculating just for the "lower triangle":</p>
  
  <pre><code>g1-&gt; 0   1   2   3
  g2
  0                    #(0,0 to 0,1-1 = nothing)
  1    X               #(1,0 to 1,1-1 = 1,0)
  2    X   X           #(2,0 to 2,2-1 = 2,0; 2,1)
  3    X   X   X       #(3,0 to 3,3-1 = 3,0; 3,1 and 3,2)
  4    X   X   X   X   #(4,0 to 4,4-1 = 4,0; 4,1; 4,2 and 4,3)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-13T09:45:46.040Z
Answers.Text: Thank you very much. It works this way.
Answers.UserDisplayName: Elyas Mohammadi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-03T21:44:12.630Z
Answers.LastActivityDate: 2020-02-03T21:44:12.630Z
Answers.UserDisplayName: Ram RS
Answers.Body: <p>Actually I found an easy and efficient way.</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import pearsonr
  from scipy.stats import spearmanr
  from scipy.stats import kendalltau
  
  pro=pd.read_csv('m_test.csv', index_col=0)
  
  xInds = []
  yInds = []
  for i in range(len(pro.index)):
      for j in range(i+1, len(pro.index)):
          xInds.append(i)
          yInds.append(j)
  
  z=0
  Rlist_sp = []
  Plist_sp = []
  Rlist_pe = []
  Plist_pe = []
  while z &lt; len(xInds):
      b=xInds[z]
      c=yInds[z]
  
      spR, spP = spearmanr(pro.iloc[b].values, pro.iloc[c].values)
      peR, peP = pearsonr(pro.iloc[b].values, pro.iloc[c].values) 
  
      Rlist_sp.append(spR)
      Plist_sp.append(spP)
      Rlist_pe.append(peR)
      Plist_pe.append(peP)    
      z=z+1 
  
  G1=pd.DataFrame(xInds)
  G2=pd.DataFrame(yInds)
  R_sp=pd.DataFrame(Rlist_sp)
  P_sp=pd.DataFrame(Plist_sp)
  R_pe=pd.DataFrame(Rlist_pe)
  P_pe=pd.DataFrame(Plist_pe)        
  
  Final=pd.concat([G1, G2, R_sp, P_sp, R_pe, P_pe], axis=1)
  np.savetxt('Final_gg_A549.txt', Final.values, fmt='%s', delimiter='\t')  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-13T09:47:33.470Z
Answers.LastActivityDate: 2020-02-13T09:47:33.470Z
Answers.UserDisplayName: Elyas Mohammadi
Body: <p>I am trying to make a paired matrix of gene-gene correlation. Considering that I have a huge matrix (13000 genes and 900 samples) and for some reasons I don't want to decrease the number of my genes, my gene-correlation matrix would be 13000*13000 and my paired matrix will become 169 million *4 (Column 1: Gene 1; Column 2 : Gene 2; Column 3: Correlations; Column 4: P-values) . In this case, I have to exclude unnecessary calculations as much as I can. I have excluded the situation that Gene 1 = Gene 2. But I couldn't find a way to exclude the condition that "Column 1: Gene 1 ; Column 2: Gene 2 = Column 1: Gene 2; Column 2: Gene 1 ". To make a long story short, correlation between G1 and G2 is equal to G2 and G1. It is like calculating just lower section of diag in a symmetric matrix. I would be grateful if anybody help me in this case. I have enclosed my python codes here:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import spearmanr
  
  
  din=pd.read_csv('m_test.csv', index_col=0)
  
  out=pd.DataFrame()
  outdf=pd.DataFrame()
  
  for g1 in din.index:
      for g2 in din.index:
          temp=din.loc[[g1, g2]]
          if g1==g2:
              next
          else: 
              spR, spP=spearmanr(temp.loc[g1], temp.loc[g2])
              frame={'g1':[g1], 'g2':[g2], 'spR':[spR], 'spP':[spP]}
  
              out=pd.DataFrame(frame)
  
              outdf=pd.concat([outdf, out])
  </code></pre>
  
Tags: <python><r><gene-expression><correlation>
Title: How to exclude the repetition of gene-gene correlation calculation in python?
ThreadText: # How to exclude the repetition of gene-gene correlation calculation in python?
  
  (Asked by: username_0 on 2020-01-31)
  
  I am trying to make a paired matrix of gene-gene correlation. Considering that I have a huge matrix (13000 genes and 900 samples) and for some reasons I don't want to decrease the number of my genes, my gene-correlation matrix would be 13000\*13000 and my paired matrix will become 169 million \*4 (Column 1: Gene 1; Column 2 : Gene 2; Column 3: Correlations; Column 4: P-values) . In this case, I have to exclude unnecessary calculations as much as I can. I have excluded the situation that Gene 1 = Gene 2. But I couldn't find a way to exclude the condition that "Column 1: Gene 1 ; Column 2: Gene 2 = Column 1: Gene 2; Column 2: Gene 1 ". To make a long story short, correlation between G1 and G2 is equal to G2 and G1. It is like calculating just lower section of diag in a symmetric matrix. I would be grateful if anybody help me in this case. I have enclosed my python codes here:
  
  ```
  import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import spearmanr
  
  din=pd.read_csv('m_test.csv', index_col=0)
  
  out=pd.DataFrame()
  outdf=pd.DataFrame()
  
  for g1 in din.index:
      for g2 in din.index:
          temp=din.loc[[g1, g2]]
          if g1==g2:
              next
          else: 
              spR, spP=spearmanr(temp.loc[g1], temp.loc[g2])
              frame={'g1':[g1], 'g2':[g2], 'spR':[spR], 'spP':[spP]}
  
              out=pd.DataFrame(frame)
  
              outdf=pd.concat([outdf, out])
  
  ```
  
  **Tags:** python, r, gene-expression, correlation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-03. Score: 1
  
  You don't need `g2` to go from `0` to `din.index`, just from `0` to `g1 - 1`. This way, you'll end up calculating just for the "lower triangle":
  
  ```
  g1-> 0   1   2   3
  g2
  0                    #(0,0 to 0,1-1 = nothing)
  1    X               #(1,0 to 1,1-1 = 1,0)
  2    X   X           #(2,0 to 2,2-1 = 2,0; 2,1)
  3    X   X   X       #(3,0 to 3,3-1 = 3,0; 3,1 and 3,2)
  4    X   X   X   X   #(4,0 to 4,4-1 = 4,0; 4,1; 4,2 and 4,3)
  
  ```
  
  > **Comments:**
  > **username_0**: Thank you very much. It works this way.  
  
  ---
  
  ### Answer by username_0 on 2020-02-13. Score: 1
  
  Actually I found an easy and efficient way.
  
  ```
  import pandas as pd
  import numpy as np
  import scipy
  import math
  import openpyxl
  from openpyxl import Workbook
  from scipy.stats import pearsonr
  from scipy.stats import spearmanr
  from scipy.stats import kendalltau
  
  pro=pd.read_csv('m_test.csv', index_col=0)
  
  xInds = []
  yInds = []
  for i in range(len(pro.index)):
      for j in range(i+1, len(pro.index)):
          xInds.append(i)
          yInds.append(j)
  
  z=0
  Rlist_sp = []
  Plist_sp = []
  Rlist_pe = []
  Plist_pe = []
  while z < len(xInds):
      b=xInds[z]
      c=yInds[z]
  
      spR, spP = spearmanr(pro.iloc[b].values, pro.iloc[c].values)
      peR, peP = pearsonr(pro.iloc[b].values, pro.iloc[c].values) 
  
      Rlist_sp.append(spR)
      Plist_sp.append(spP)
      Rlist_pe.append(peR)
      Plist_pe.append(peP)    
      z=z+1 
  
  G1=pd.DataFrame(xInds)
  G2=pd.DataFrame(yInds)
  R_sp=pd.DataFrame(Rlist_sp)
  P_sp=pd.DataFrame(Plist_sp)
  R_pe=pd.DataFrame(Rlist_pe)
  P_pe=pd.DataFrame(Plist_pe)        
  
  Final=pd.concat([G1, G2, R_sp, P_sp, R_pe, P_pe], axis=1)
  np.savetxt('Final_gg_A549.txt', Final.values, fmt='%s', delimiter='\t')  
  
  ```
  
  
  ---
  
  