Answers.Body: <p>An <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4720993/" rel="nofollow noreferrer">UpSet Plot</a> is a useful way of visualising these kind of multi-dimensional overlaps. There's an <a href="https://cran.r-project.org/web/packages/UpSetR/README.html" rel="nofollow noreferrer">R package</a> for producing them, with a reasonable set of examples. </p>
  
  <p>The input is a binary matrix indicating set membership:</p>
  
  <pre><code>         h2    h4    h6...
  GeneA     0     1     0
  GeneB     1     1     0
  GeneC     0     1     1
  ...
  </code></pre>
  
  <p>And the output is a pair of bar charts, showing the size of various sets, with a dot matrix showing the set membership. An example from the UpSetR documentation:</p>
  
  <pre><code>library(UpSetR)
  mutations &lt;- read.csv(system.file("extdata", "mutations.csv", 
                        package = "UpSetR"), header=T, sep = ",")
  upset(mutations, sets = c("PTEN", "TP53", "EGFR", "PIK3R1", "RB1"),
        sets.bar.color = "#56B4E9", order.by = "freq", 
        empty.intersections = "on")
  </code></pre>
  
  <p><a href="https://i.sstatic.net/wSfK4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wSfK4.png" alt="Example UpSet Plot"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-26T14:41:21.180Z
Answers.LastActivityDate: 2018-06-26T14:41:21.180Z
Answers.UserDisplayName: sjcockell
Answers.Body: <p>I would simply do this...
  </p>
  
  <pre><code># sepcify your timepoints, each timepoint is a list of genes!!
  timepoints &lt;- list(h2,h4,h6,h8, h10,h12,h14,h16)
  # initialize an empty array that you will fill with your timepoints
  vect = vector()
  # go over all combinations and fill the array with the set intersection values you want (I guess...)
  for (i in timepoints) {
    for (j in timepoints) {
      vect &lt;- c(vect, length(intersect(i,j))/length(i))}}
  # reshape your array to a more ordered matrix  
  your_matrix &lt;- matrix(vect, nrow=length(timepoints))
  # assign columns and rows names
  colnames(your_matrix) &lt;- c("h2","h4","h6","h8","h10","h12","h14","h16")
  rownames(your_matrix) &lt;- c("h2","h4","h6","h8","h10","h12","h14","h16")
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-24T18:42:58.863Z
Answers.LastActivityDate: 2018-06-24T18:42:58.863Z
Answers.UserDisplayName: aerijman
Answers.Body: <p>EDIT: Sorry, I did not see the R tag and wrote the code in bash. I still left the bash example below and added an R version.</p>
  
  <p>Depends on what you want to do, but there are two common ways in which you could store this. Either as wide format: </p>
  
  <pre><code>      h2  h4  h6 ...
  Gene1 1   1   0
  Gene2 0   1   1
  Gene3 1   1   1
  ...
  </code></pre>
  
  <p>Where 1 denotes presence and 0 absence, or as long format:</p>
  
  <pre><code>      timepoint
  Gene1 h2
  Gene1 h4
  Gene2 h4
  Gene2 h6
  Gene3 h2
  Gene3 h4
  Gene3 h6
  ...
  </code></pre>
  
  <p>The best format probably depends on what you want to do. Since you did not specify a method/language, I took the liberty to write a few lines of bash to go from the gene lists to the matrices. 
  Bash examples assume your files are named h2, h4 etc and are tab separated with gene names in the first column. R examples assume <code>tables</code> is a list of your data frames and <code>timepoints</code> is a vector <code>c("h2", "h4", ...)</code>.</p>
  
  <p><strong>Generate table in wide format:</strong></p>
  
  <ol>
  <li>Concatenate gene names of all tables to a separate file</li>
  <li>Extract unique values in this file</li>
  <li>Perform outer join between the separate file and each table</li>
  </ol>
  
  <p><em>R:</em></p>
  
  <pre><code>all_names &lt;- c()
  for (tp in timepoints){
    all_names &lt;- append(all_names, tables[[tp]]$genes)
  }
  all_names &lt;- unique(all_names)
  
  wide_table &lt;- matrix(nrow=length(all_names), ncol=(1+length(timepoints)))
  wide_table &lt;- as.data.frame(wide_table)
  colnames(wide_table) &lt;- append("genes", timepoints)
  
  for (tp in timepoints){
   tmp &lt;- tables[[tp]]
   tmp$tp &lt;- 1
   tmp &lt;- merge(wide_table, tmp, by="genes", all=T)
   wide_table$tp &lt;- tmp$tp
  }
  </code></pre>
  
  <p><em>Bash:</em></p>
  
  <p>Note: you will might need to add headers to this table</p>
  
  <pre><code>for timefile in $yourfiles;do
    # make a file with all gene names
    awk -v tp=$timefile '{print $1}' &gt;&gt; names_file
  done
  
  sort names_file | uniq &gt; tmp &amp;&amp; mv tmp names_file
  
  for timefile in $yourfiles;do
    join -a1 -a2  -j 1 -o 0,1.2,2.2 -e 0 $names_file &lt;(awk '{print$1,1}' $timefile | sort)
  done
  </code></pre>
  
  <p><strong>Generate table in long format:</strong></p>
  
  <ol>
  <li>For each table, add a column with timepoint</li>
  <li>Concatenate all tables</li>
  </ol>
  
  <p><em>R:</em></p>
  
  <pre><code>df &lt;- data.frame()
  for (tp in timepoints){
    tables[[tp]][[timepoint]] &lt;- tp
    df &lt;- rbind(df, tables[[tp]] )
  }
  </code></pre>
  
  <p><em>Bash:</em></p>
  
  <pre><code>   for timefile in $yourfiles;do
        awk -v tp=$timefile '{print $1,timefile}' &gt;&gt; long_table
      done 
  </code></pre>
  
  <p>EDIT2: To follow up on this, here is a simple way (using the package <a href="https://www.tidyverse.org/" rel="nofollow noreferrer">dplyr</a>) to count the number of common genes between all possible combination of time points. Assuming you have your data in long format:</p>
  
  <pre><code>df &lt;- data.frame(genes=c("G1","G1","G2","G3","G3"), tp=c("h2","h4","h2","h4","h6"))
  
    genes tp
  1    G1 h2
  2    G1 h4
  3    G2 h2
  4    G3 h4
  5    G3 h6
  </code></pre>
  
  <p>You can get the co-occurrences using :</p>
  
  <pre><code>library(dplyr)
  # Make timepoint ordinal factor
  df &lt;- mutate(df, tp = factor(tp, ordered=T))
  df &lt;- df %&gt;% 
    # Compute all pairwise co-occurrences
    full_join(df, by="genes") %&gt;% 
    group_by(tp.x, tp.y) %&gt;% 
    summarise(N_genes = length(unique(genes))) %&gt;%
    # Exclude self pairs (gene in h2 and h2)
    filter(tp.x != tp.y) %&gt;% 
    # Check for duplicate pairs in reverse order (h2-h4 and h4-h2)
    mutate(comb=ifelse(tp.x &lt; tp.y, 
                       yes = paste(tp.x, tp.y), 
                       no  = paste(tp.y, tp.x)))
  # Remove the duplicate pairs
  df &lt;- df[which(!duplicated(df$comb)),]
  </code></pre>
  
  <p>And visualise it using for example a heatmap:
  (ofc it looks stupid because I have a tiny sample where not all combinations are even present)</p>
  
  <pre><code># Visualize number of genes present in each combination
  library(ggplot2)
  ggplot(df, aes(x=tp.x, y=tp.y, fill=N_genes)) + geom_tile()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T17:41:29.120Z
Answers.Text: excuse me, i did not quite understand. i have 8 separate lists of genes so how i know about 0 and 1 (absence and presence of a genes among the lists)?
Answers.UserDisplayName: Zizogolu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T17:45:49.877Z
Answers.Text: @FereshTeh 0 and 1 were just placeholder values to give the info "geneY is present (1) / absent (0) at timepoint hX" . You could have expression values, ratios, pvalues or whatever variable instead of 1, and NA instead of 0, for example. Does it make sense ?
Answers.UserDisplayName: cmdoret
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T18:10:39.173Z
Answers.Text: thanks a lot, actually i just have 8 lists of genes, lets say that each list has one column of genes. i want a pairwise comparison for common genes for any possible combinations of these time points.
Answers.UserDisplayName: Zizogolu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T18:28:31.267Z
Answers.Text: Oh sorry, i did not understand the last part about pairwise comparisons. First, if you want to filter common genes, you could just filter rows (genes) which have values available for more than x timepoints. For pairwise comparison, maybe that'd be easier if you provided more details/background. (E.g. do you want to compute a single summary stat per condition and compute it, or is it per gene ?)
Answers.UserDisplayName: cmdoret
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T20:36:07.313Z
Answers.Text: thanks a lot, i think per condition. actually i need common genes. for instance i have 630 common genes between h2 and h4 time points, while i have 1143 genes in h2 and 768 genes in h4 itself. so, i have two ways comparison; 630/1143 and 630/768. that would be great if i could do the same for all of my time points. something like having a matrix of percentage or a graph of them.
Answers.UserDisplayName: Zizogolu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T21:21:57.033Z
Answers.Text: Oh I think I understand. Could you update your original question with the exact comparisons you want to make (e.g. Compute number of available genes between all possible pairs of combination). Then I can make sure I got it right and update my response.
Answers.UserDisplayName: cmdoret
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T22:49:24.987Z
Answers.Text: I updated my solution, please mark it as accepted if it answers your question.
Answers.UserDisplayName: cmdoret
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T10:45:53.337Z
Answers.Text: excuse me, i quite did not understand if i have 9 files each with one column of genes, how i can have df (long format). :(:(
Answers.UserDisplayName: Zizogolu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T11:25:58.760Z
Answers.Text: I slightly edited my answer. hopefully it is more clear. To generate df, you need to generate the table in long format. I made a section with instructions (bash and R) to generate such table. If you chose R, you might need to edit the code to suite your data. If you chose bash, you just need to read the "long_table" file into R using read.table().
Answers.UserDisplayName: cmdoret
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T11:53:04.160Z
Answers.Text: sorry for sure your code will work. i am really sorry but I'm too new in coding. for example for producing long format n r, should i have a df for each time point? i mean 9 df files? i don't know what is tp or should i put my 9 columns in  one file with its own headers for example h2, h4 in header of each column. i really could not figure out. however definitely your code is so practical
Answers.UserDisplayName: Zizogolu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T12:55:42.927Z
Answers.Text: No problem with being new to programming. I am afraid I cannot help you anymore than that, but if you get a bit more into programming first, I think it will make sense. There are great free resources out there. Here is a general purpose list:https://github.com/Michael0x2a/curated-programming-resources/blob/master/resources.md 
  For R I would maybe try: https://www.datacamp.com/
  For python in bioinformatics: http://rosalind.info/problems/locations/
  For bash: https://www.codecademy.com
  Hopefully some of these will be helpful to you :) Best of luck
Answers.UserDisplayName: cmdoret
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T16:09:13.220Z
Answers.LastActivityDate: 2018-06-25T07:35:24.700Z
Answers.LastEditDate: 2018-06-25T07:35:24.700Z
Answers.UserDisplayName: cmdoret
Body: <p>I have 8 lists of differentially expressed genes for 8 time points (2, 4, 6, 8, 10, 12, 14, 16 hours), the number of genes in each of these lists are not equal.</p>
  
  <p>For example there are 630 common genes between h2 and h4 time points but I have 1143 genes in h2 list and 768 genes in h4 list , so common genes in h2 vs h4 would be 630/1143 and common genes h4 vs h2 would be 630/768.</p>
  
  <p>That would would be great if I have a code to produce a matrix or plot of all possible pairwise combinations of my 8 time points.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-21T10:03:48.153Z
Comments.Text: cross posted https://www.biostars.org/p/321861/
Comments.UserDisplayName: Pierre
Tags: <r>
Title: Multiway intersection of 8 lists of genes
ThreadText: # Multiway intersection of 8 lists of genes
  
  (Asked by: username_0 on 2018-06-21)
  
  I have 8 lists of differentially expressed genes for 8 time points (2, 4, 6, 8, 10, 12, 14, 16 hours), the number of genes in each of these lists are not equal.
  
  For example there are 630 common genes between h2 and h4 time points but I have 1143 genes in h2 list and 768 genes in h4 list , so common genes in h2 vs h4 would be 630/1143 and common genes h4 vs h2 would be 630/768.
  
  That would would be great if I have a code to produce a matrix or plot of all possible pairwise combinations of my 8 time points.
  
  **Tags:** r
  
  > **Comments:**
  > **username_1**: cross posted https://www.biostars.org/p/321861/  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2018-06-21. Score: 3 | Accepted answer
  
  EDIT: Sorry, I did not see the R tag and wrote the code in bash. I still left the bash example below and added an R version.
  
  Depends on what you want to do, but there are two common ways in which you could store this. Either as wide format:
  
  ```
        h2  h4  h6 ...
  Gene1 1   1   0
  Gene2 0   1   1
  Gene3 1   1   1
  ...
  
  ```
  
  Where 1 denotes presence and 0 absence, or as long format:
  
  ```
        timepoint
  Gene1 h2
  Gene1 h4
  Gene2 h4
  Gene2 h6
  Gene3 h2
  Gene3 h4
  Gene3 h6
  ...
  
  ```
  
  The best format probably depends on what you want to do. Since you did not specify a method/language, I took the liberty to write a few lines of bash to go from the gene lists to the matrices.
  Bash examples assume your files are named h2, h4 etc and are tab separated with gene names in the first column. R examples assume `tables` is a list of your data frames and `timepoints` is a vector `c("h2", "h4", ...)`.
  
  **Generate table in wide format:**
  
  1. Concatenate gene names of all tables to a separate file
  2. Extract unique values in this file
  3. Perform outer join between the separate file and each table
  
  *R:*
  
  ```
  all_names <- c()
  for (tp in timepoints){
    all_names <- append(all_names, tables[[tp]]$genes)
  }
  all_names <- unique(all_names)
  
  wide_table <- matrix(nrow=length(all_names), ncol=(1+length(timepoints)))
  wide_table <- as.data.frame(wide_table)
  colnames(wide_table) <- append("genes", timepoints)
  
  for (tp in timepoints){
   tmp <- tables[[tp]]
   tmp$tp <- 1
   tmp <- merge(wide_table, tmp, by="genes", all=T)
   wide_table$tp <- tmp$tp
  }
  
  ```
  
  *Bash:*
  
  Note: you will might need to add headers to this table
  
  ```
  for timefile in $yourfiles;do
    # make a file with all gene names
    awk -v tp=$timefile '{print $1}' >> names_file
  done
  
  sort names_file | uniq > tmp && mv tmp names_file
  
  for timefile in $yourfiles;do
    join -a1 -a2  -j 1 -o 0,1.2,2.2 -e 0 $names_file <(awk '{print$1,1}' $timefile | sort)
  done
  
  ```
  
  **Generate table in long format:**
  
  1. For each table, add a column with timepoint
  2. Concatenate all tables
  
  *R:*
  
  ```
  df <- data.frame()
  for (tp in timepoints){
    tables[[tp]][[timepoint]] <- tp
    df <- rbind(df, tables[[tp]] )
  }
  
  ```
  
  *Bash:*
  
  ```
     for timefile in $yourfiles;do
        awk -v tp=$timefile '{print $1,timefile}' >> long_table
      done 
  
  ```
  
  EDIT2: To follow up on this, here is a simple way (using the package [dplyr](https://www.tidyverse.org/)) to count the number of common genes between all possible combination of time points. Assuming you have your data in long format:
  
  ```
  df <- data.frame(genes=c("G1","G1","G2","G3","G3"), tp=c("h2","h4","h2","h4","h6"))
  
    genes tp
  1    G1 h2
  2    G1 h4
  3    G2 h2
  4    G3 h4
  5    G3 h6
  
  ```
  
  You can get the co-occurrences using :
  
  ```
  library(dplyr)
  # Make timepoint ordinal factor
  df <- mutate(df, tp = factor(tp, ordered=T))
  df <- df %>% 
    # Compute all pairwise co-occurrences
    full_join(df, by="genes") %>% 
    group_by(tp.x, tp.y) %>% 
    summarise(N_genes = length(unique(genes))) %>%
    # Exclude self pairs (gene in h2 and h2)
    filter(tp.x != tp.y) %>% 
    # Check for duplicate pairs in reverse order (h2-h4 and h4-h2)
    mutate(comb=ifelse(tp.x < tp.y, 
                       yes = paste(tp.x, tp.y), 
                       no  = paste(tp.y, tp.x)))
  # Remove the duplicate pairs
  df <- df[which(!duplicated(df$comb)),]
  
  ```
  
  And visualise it using for example a heatmap:
  (ofc it looks stupid because I have a tiny sample where not all combinations are even present)
  
  ```
  # Visualize number of genes present in each combination
  library(ggplot2)
  ggplot(df, aes(x=tp.x, y=tp.y, fill=N_genes)) + geom_tile()
  
  ```
  
  > **Comments:**
  > **username_0**: excuse me, i did not quite understand. i have 8 separate lists of genes so how i know about 0 and 1 (absence and presence of a genes among the lists)?  
  > **username_4**: @FereshTeh 0 and 1 were just placeholder values to give the info "geneY is present (1) / absent (0) at timepoint hX" . You could have expression values, ratios, pvalues or whatever variable instead of 1, and NA instead of 0, for example. Does it make sense ?  
  > **username_0**: thanks a lot, actually i just have 8 lists of genes, lets say that each list has one column of genes. i want a pairwise comparison for common genes for any possible combinations of these time points.  
  > **username_4**: Oh sorry, i did not understand the last part about pairwise comparisons. First, if you want to filter common genes, you could just filter rows (genes) which have values available for more than x timepoints. For pairwise comparison, maybe that'd be easier if you provided more details/background. (E.g. do you want to compute a single summary stat per condition and compute it, or is it per gene ?)  
  > **username_0**: thanks a lot, i think per condition. actually i need common genes. for instance i have 630 common genes between h2 and h4 time points, while i have 1143 genes in h2 and 768 genes in h4 itself. so, i have two ways comparison; 630/1143 and 630/768. that would be great if i could do the same for all of my time points. something like having a matrix of percentage or a graph of them.  
  > **username_4**: Oh I think I understand. Could you update your original question with the exact comparisons you want to make (e.g. Compute number of available genes between all possible pairs of combination). Then I can make sure I got it right and update my response.  
  > **username_4**: I updated my solution, please mark it as accepted if it answers your question.  
  > **username_0**: excuse me, i quite did not understand if i have 9 files each with one column of genes, how i can have df (long format). :(:(  
  > **username_4**: I slightly edited my answer. hopefully it is more clear. To generate df, you need to generate the table in long format. I made a section with instructions (bash and R) to generate such table. If you chose R, you might need to edit the code to suite your data. If you chose bash, you just need to read the "long_table" file into R using read.table().  
  > **username_0**: sorry for sure your code will work. i am really sorry but I'm too new in coding. for example for producing long format n r, should i have a df for each time point? i mean 9 df files? i don't know what is tp or should i put my 9 columns in  one file with its own headers for example h2, h4 in header of each column. i really could not figure out. however definitely your code is so practical  
  > **username_4**: No problem with being new to programming. I am afraid I cannot help you anymore than that, but if you get a bit more into programming first, I think it will make sense. There are great free resources out there. Here is a general purpose list:https://github.com/Michael0x2a/curated-programming-resources/blob/master/resources.md  For R I would maybe try: https://www.datacamp.com/ For python in bioinformatics: http://rosalind.info/problems/locations/ For bash: https://www.codecademy.com Hopefully some of these will be helpful to you :) Best of luck  
  
  ---
  
  ### Answer by username_2 on 2018-06-26. Score: 3
  
  An [UpSet Plot](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4720993/) is a useful way of visualising these kind of multi-dimensional overlaps. There's an [R package](https://cran.r-project.org/web/packages/UpSetR/README.html) for producing them, with a reasonable set of examples.
  
  The input is a binary matrix indicating set membership:
  
  ```
           h2    h4    h6...
  GeneA     0     1     0
  GeneB     1     1     0
  GeneC     0     1     1
  ...
  
  ```
  
  And the output is a pair of bar charts, showing the size of various sets, with a dot matrix showing the set membership. An example from the UpSetR documentation:
  
  ```
  library(UpSetR)
  mutations <- read.csv(system.file("extdata", "mutations.csv", 
                        package = "UpSetR"), header=T, sep = ",")
  upset(mutations, sets = c("PTEN", "TP53", "EGFR", "PIK3R1", "RB1"),
        sets.bar.color = "#56B4E9", order.by = "freq", 
        empty.intersections = "on")
  
  ```
  
  [![Example UpSet Plot](https://i.sstatic.net/wSfK4.png)](https://i.sstatic.net/wSfK4.png)
  
  
  ---
  
  ### Answer by username_3 on 2018-06-24. Score: 1
  
  I would simply do this...
  
  ```
  # sepcify your timepoints, each timepoint is a list of genes!!
  timepoints <- list(h2,h4,h6,h8, h10,h12,h14,h16)
  # initialize an empty array that you will fill with your timepoints
  vect = vector()
  # go over all combinations and fill the array with the set intersection values you want (I guess...)
  for (i in timepoints) {
    for (j in timepoints) {
      vect <- c(vect, length(intersect(i,j))/length(i))}}
  # reshape your array to a more ordered matrix  
  your_matrix <- matrix(vect, nrow=length(timepoints))
  # assign columns and rows names
  colnames(your_matrix) <- c("h2","h4","h6","h8","h10","h12","h14","h16")
  rownames(your_matrix) <- c("h2","h4","h6","h8","h10","h12","h14","h16")
  
  ```
  
  
  ---
  
  