Answers.Body: <h3>Files in a cellranger run</h3>
  
  <p>Cellranger removes the intermediary files after the clusters have been computed. This can be observed in the following path from the cellranger project directory. This contains empty directories.</p>
  
  <pre><code>SC_RNA_AGGREGATOR_CS/SC_RNA_ANALYZER/RUN_GRAPH_CLUSTERING
  </code></pre>
  
  <p>However, the clusters generated by cellranger can be found in the file:</p>
  
  <pre><code>outs/analysis/clustering/graphclust/clusters.csv
  </code></pre>
  
  <p>Clusters can also be "exported" from <code>outs/cloupe.cloupe</code> file into a CSV using the Cloupe Browser. These CSVs can be imported and used for plotting in R or Python.</p>
  
  <p>The graph structure used to compute these (with the Louvian algorithm) cannot be found in the files generated by cellranger.</p>
  
  <h3>Reproducing graph-based clustering</h3>
  
  <p>The Suerat R package performs graph-based clustering. The shared nearest neighbour (SNN) modularity optimisation uses the same parameters as the Python scripts called by cellranger (<code>k=30</code>). This generates an SNN matrix which can be used to generate a graph structure from an adjacency matrix. For a Seurat object <code>test</code> the SNN sparse matrix can be extract with:</p>
  
  <pre><code>FindClusters(test, dims.use = 1:20, algorithm = 1,
               resolution = 0.6, k.param = 30, save.SNN = TRUE,
               reduction.use = "pca", metric = "euclidean")
  SNN_Matrix &lt;- as.matrix(ceiling(as.matrix(test@snn)
  graph_structure &lt;- igraph::graph_from_adjacency(SNN_matrix)
  </code></pre>
  
  <p><code>FindClusters</code> from the <code>Seurat</code> package requires a Seurat object that has already has a dimension reduction run on it such as <code>RunPCA</code>. The <code>algorithm = 1</code> is the Louvian algorithm as performed by cellranger. The <code>algorithm</code>, <code>reduction.use</code> and <code>metric</code> can be changed as mentioned in the <code>Suerat</code> package documentation. Clustering algorithms not yet supported by Seurat can be run on the <code>graph_structure</code> resulting from extracting the SNN matrix. It is not necessary to compute the edges separately, the Seurat functions will compute and return these.</p>
  
  <p>For example, the Leiden algorithm can be computed as follows:</p>
  
  <pre><code>devtools:install_github("TomKellyGenetics/leiden")
  library(leiden)
  clusters &lt;- leiden(SNN_Matrix)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T10:09:46.463Z
Answers.LastActivityDate: 2018-11-15T10:09:46.463Z
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.Body: <p><strong>Cell Ranger</strong></p>
  
  <p>You can't download the tSNE coordinates for cells directly from the <em>Analysis</em> tab of the fancy, polished .html document that Cell Ranger produces. If you have access to the machine on which the pipeline was run, you can grab the intermediate files. Everything is <a href="https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/analysis" rel="nofollow noreferrer">well documented</a> by 10X. They explain defaults etc. on that page. The tSNE coordinates are available at:</p>
  
  <pre><code>analysis/tsne/2_components/projection.csv
  </code></pre>
  
  <p>and the contents look like:</p>
  
  <pre><code>$ head -5 analysis/tsne/2_components/projection.csv
  Barcode,TSNE-1,TSNE-2
  AAACATACAACGAA-1,-13.5494,1.4674
  AAACATACTACGCA-1,-2.7325,-10.6347
  AAACCGTGTCTCGC-1,12.9590,-1.6369
  AAACGCACAACCAC-1,-9.3585,-6.7300
  </code></pre>
  
  <p><strong>Seurat</strong></p>
  
  <p>And the second point: Cell Ranger is replicating the Seurat workflow. Seurat uses the Louvain method for community detection by default. Check out the <a href="https://satijalab.org/seurat/pbmc3k_tutorial.html" rel="nofollow noreferrer">intro tutorial</a> and see if you can follow along. This tutorial works with 10X data, in fact the first function call in the tutorial reads in the 10X matrices. If you are already plotting in R you'll speed through this tutorial. Seurat is great to work with, you'll have tSNEs and clusters in no time. And there's so much more you can do from there!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-13T23:31:58.013Z
Answers.Text: Yes I’m familiar with the Seurat workflow. I don’t understand the Python source code well enough to know which parameters were used by cellranger.
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T05:21:02.980Z
Answers.Text: If you're unwilling to use a different software package, why mention that as a possibility in your second point? The matrices are very lightly analyzed data. The unfiltered matrices are counts for all cell barcodes. The filtered matrices are count for cell arcodes determined to come from real cells. You still have to normalize and perform quality control filtering on this data efore it's ready for its first round of dimensional reduction and clustering.
Answers.UserDisplayName: Kohl Kinning
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T05:23:23.187Z
Answers.Text: If you want information about their algorithms, see [here](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/algorithms/overview). There is also info on that page about the matrices. This should help with your understanding of how Cell Ranger is massaging the data.
Answers.UserDisplayName: Kohl Kinning
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T05:27:58.470Z
Answers.Text: I’m considering whether it is necessary to compute the graph again of it has already been done by cellranger. If it has not been saved by this program, it will be needed to compute it as you’ve suggested. In that case, more detailed information will be helpful. I have (and will continue) to do it this way but it would be reassuring to know I’m on the right track and not reinventing a graph structure that I already have saved to disk somewhere.
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T21:50:51.213Z
Answers.Text: Ah, I was thinking you just just had the final product. If you have access to the machine where the pipeline was run you can get the tSNE coordinates.
Answers.UserDisplayName: Kohl Kinning
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T22:08:58.240Z
Answers.Text: Yes I ran the cellranger pipeline myself. However, I’m not looking for the tSNE coordinates. I’m looking for a list of edges used for the graph-based clustering (I can already find the clusters).
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T10:21:01.627Z
Answers.Text: I've managed to get want I needed with the `Seurat::FindClusters` function from the `object@snn` output. I've detailed my process below.
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-13T22:54:35.590Z
Answers.LastActivityDate: 2018-11-14T21:56:16.010Z
Answers.LastEditDate: 2018-11-14T21:56:16.010Z
Answers.UserDisplayName: Kohl Kinning
Body: <p>I’ve run the cellranger analysis pipeline on single cell RNASeq datasets. I can import the matrix and graph-based clusters into R. Doing this I can optimise the dimension reduction and plot cells with coloured by clusters generated by cellranger. I wish to optimise the graph-based clustering as well.</p>
  
  <p>Is it possible to either:</p>
  
  <ul>
  <li><p>obtain the graph generated by cellranger from the files generated. (I can only find the final clusters in the analysis output)</p></li>
  <li><p>reproduce generating a similar graph (used for clustering) from the output matrix (preferably with R or Python.</p></li>
  </ul>
  
  <p>With either approach, I am looking for a list of edges for nearest neighbors in the gene-barcode UMI matrix to use as input for a clustering algorithm.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-13T11:26:25.637Z
Comments.Text: Could you provide some data as an example or pick the data from the vignette to answer you would work for you? What have you tried to get the graph or reproduce it?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-13T13:39:19.730Z
Comments.Text: I’ve tried several ways to reproduce this graph in R but I’m unsure that I understand well enough what cellranger is doing (based on Python code in the GitHub repo) to generate inputs for the Louvian algorithm. I’m now unsure that this is necessary since any cellranger run generates various output files for the analysis in a specific directory structure. I think this question requires someone with familiar with this software already and has run it themselves. I cannot provide the data that I’m working with currently and the files are too large even if I could.
Comments.UserDisplayName: Tom Kelly ケリー・トム
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-13T15:34:04.823Z
Comments.Text: Hi @TomKelly. I don't know if I understood exactly the question. However, you may find useful to use [scanpy](https://scanpy.readthedocs.io/en/latest/) for importing the data, run the Louvain clustering, and plot the data with the clusters. That is a Python module.
Comments.UserDisplayName: gc5
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-13T16:43:33.513Z
Comments.Text: @TomKelly Could you provide your attempts and why they fail? This would make it easier for me to avoid this pitfalls (if I try to answer your question)
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-13T20:10:26.687Z
Comments.Text: I don't understand what plot you want. You know where the T-SNE coordinates are, right?
Comments.UserDisplayName: swbarnes2
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-13T23:30:30.043Z
Comments.Text: No, I can perform tSNE in R. I want the graph that the graph-based clusters are derived from. I want to annotate clusters into it. Can these be found in the output data or does it have to be computed from the matrix?
Comments.UserDisplayName: Tom Kelly ケリー・トム
Tags: <scrnaseq><single-cell><clustering><10x-genomics><cellranger>
Title: Can I get the graph generated by cellranger
ThreadText: # Can I get the graph generated by cellranger
  
  (Asked by: username_0 on 2018-11-13)
  
  I’ve run the cellranger analysis pipeline on single cell RNASeq datasets. I can import the matrix and graph-based clusters into R. Doing this I can optimise the dimension reduction and plot cells with coloured by clusters generated by cellranger. I wish to optimise the graph-based clustering as well.
  
  Is it possible to either:
  
  * obtain the graph generated by cellranger from the files generated. (I can only find the final clusters in the analysis output)
  * reproduce generating a similar graph (used for clustering) from the output matrix (preferably with R or Python.
  
  With either approach, I am looking for a list of edges for nearest neighbors in the gene-barcode UMI matrix to use as input for a clustering algorithm.
  
  **Tags:** scrnaseq, single-cell, clustering, 10x-genomics, cellranger
  
  > **Comments:**
  > **username_1**: Could you provide some data as an example or pick the data from the vignette to answer you would work for you? What have you tried to get the graph or reproduce it?  
  > **username_0**: I’ve tried several ways to reproduce this graph in R but I’m unsure that I understand well enough what cellranger is doing (based on Python code in the GitHub repo) to generate inputs for the Louvian algorithm. I’m now unsure that this is necessary since any cellranger run generates various output files for the analysis in a specific directory structure. I think this question requires someone with familiar with this software already and has run it themselves. I cannot provide the data that I’m working with currently and the files are too large even if I could.  
  > **username_2**: Hi @TomKelly. I don't know if I understood exactly the question. However, you may find useful to use [scanpy](https://scanpy.readthedocs.io/en/latest/) for importing the data, run the Louvain clustering, and plot the data with the clusters. That is a Python module.  
  > **username_1**: @TomKelly Could you provide your attempts and why they fail? This would make it easier for me to avoid this pitfalls (if I try to answer your question)  
  > **username_3**: I don't understand what plot you want. You know where the T-SNE coordinates are, right?  
  > **username_0**: No, I can perform tSNE in R. I want the graph that the graph-based clusters are derived from. I want to annotate clusters into it. Can these be found in the output data or does it have to be computed from the matrix?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2018-11-13. Score: 2 | Accepted answer
  
  **Cell Ranger**
  
  You can't download the tSNE coordinates for cells directly from the *Analysis* tab of the fancy, polished .html document that Cell Ranger produces. If you have access to the machine on which the pipeline was run, you can grab the intermediate files. Everything is [well documented](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/analysis) by 10X. They explain defaults etc. on that page. The tSNE coordinates are available at:
  
  ```
  analysis/tsne/2_components/projection.csv
  
  ```
  
  and the contents look like:
  
  ```
  $ head -5 analysis/tsne/2_components/projection.csv
  Barcode,TSNE-1,TSNE-2
  AAACATACAACGAA-1,-13.5494,1.4674
  AAACATACTACGCA-1,-2.7325,-10.6347
  AAACCGTGTCTCGC-1,12.9590,-1.6369
  AAACGCACAACCAC-1,-9.3585,-6.7300
  
  ```
  
  **Seurat**
  
  And the second point: Cell Ranger is replicating the Seurat workflow. Seurat uses the Louvain method for community detection by default. Check out the [intro tutorial](https://satijalab.org/seurat/pbmc3k_tutorial.html) and see if you can follow along. This tutorial works with 10X data, in fact the first function call in the tutorial reads in the 10X matrices. If you are already plotting in R you'll speed through this tutorial. Seurat is great to work with, you'll have tSNEs and clusters in no time. And there's so much more you can do from there!
  
  > **Comments:**
  > **username_0**: Yes I’m familiar with the Seurat workflow. I don’t understand the Python source code well enough to know which parameters were used by cellranger.  
  > **username_4**: If you're unwilling to use a different software package, why mention that as a possibility in your second point? The matrices are very lightly analyzed data. The unfiltered matrices are counts for all cell barcodes. The filtered matrices are count for cell arcodes determined to come from real cells. You still have to normalize and perform quality control filtering on this data efore it's ready for its first round of dimensional reduction and clustering.  
  > **username_4**: If you want information about their algorithms, see [here](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/algorithms/overview). There is also info on that page about the matrices. This should help with your understanding of how Cell Ranger is massaging the data.  
  > **username_0**: I’m considering whether it is necessary to compute the graph again of it has already been done by cellranger. If it has not been saved by this program, it will be needed to compute it as you’ve suggested. In that case, more detailed information will be helpful. I have (and will continue) to do it this way but it would be reassuring to know I’m on the right track and not reinventing a graph structure that I already have saved to disk somewhere.  
  > **username_4**: Ah, I was thinking you just just had the final product. If you have access to the machine where the pipeline was run you can get the tSNE coordinates.  
  > **username_0**: Yes I ran the cellranger pipeline myself. However, I’m not looking for the tSNE coordinates. I’m looking for a list of edges used for the graph-based clustering (I can already find the clusters).  
  > **username_0**: I've managed to get want I needed with the `Seurat::FindClusters` function from the `object@snn` output. I've detailed my process below.  
  
  ---
  
  ### Answer by username_0 on 2018-11-15. Score: 0
  
  ### Files in a cellranger run
  
  Cellranger removes the intermediary files after the clusters have been computed. This can be observed in the following path from the cellranger project directory. This contains empty directories.
  
  ```
  SC_RNA_AGGREGATOR_CS/SC_RNA_ANALYZER/RUN_GRAPH_CLUSTERING
  
  ```
  
  However, the clusters generated by cellranger can be found in the file:
  
  ```
  outs/analysis/clustering/graphclust/clusters.csv
  
  ```
  
  Clusters can also be "exported" from `outs/cloupe.cloupe` file into a CSV using the Cloupe Browser. These CSVs can be imported and used for plotting in R or Python.
  
  The graph structure used to compute these (with the Louvian algorithm) cannot be found in the files generated by cellranger.
  
  ### Reproducing graph-based clustering
  
  The Suerat R package performs graph-based clustering. The shared nearest neighbour (SNN) modularity optimisation uses the same parameters as the Python scripts called by cellranger (`k=30`). This generates an SNN matrix which can be used to generate a graph structure from an adjacency matrix. For a Seurat object `test` the SNN sparse matrix can be extract with:
  
  ```
  FindClusters(test, dims.use = 1:20, algorithm = 1,
               resolution = 0.6, k.param = 30, save.SNN = TRUE,
               reduction.use = "pca", metric = "euclidean")
  SNN_Matrix <- as.matrix(ceiling(as.matrix(test@snn)
  graph_structure <- igraph::graph_from_adjacency(SNN_matrix)
  
  ```
  
  `FindClusters` from the `Seurat` package requires a Seurat object that has already has a dimension reduction run on it such as `RunPCA`. The `algorithm = 1` is the Louvian algorithm as performed by cellranger. The `algorithm`, `reduction.use` and `metric` can be changed as mentioned in the `Suerat` package documentation. Clustering algorithms not yet supported by Seurat can be run on the `graph_structure` resulting from extracting the SNN matrix. It is not necessary to compute the edges separately, the Seurat functions will compute and return these.
  
  For example, the Leiden algorithm can be computed as follows:
  
  ```
  devtools:install_github("TomKellyGenetics/leiden")
  library(leiden)
  clusters <- leiden(SNN_Matrix)
  
  ```
  
  
  ---
  
  