Answers.Body: <p>You should be able to parse out what you need using the tags in the .bam.  10xGenomics' website says what tags they add.</p>
  
  <p><a href="https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/bam" rel="nofollow noreferrer">https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/bam</a></p>
  
  <p>Going backwards would also involve parsing the tags, because you have to make two fastq files, and a simple bam -> fastq pipeline won't do that correctly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T22:13:07.163Z
Answers.Text: Could you be more specific please? I'm very new to this procedure and bash scripting in general.
Answers.UserDisplayName: h3ab74
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T16:45:30.523Z
Answers.Text: Yes, 10xGenomics single cell bams have the genes called in the bam tags.  https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/bam And no, samtools fastq will not make paired end fastqs suitable to rerun through cellranger, because the read data from the read containing the cell barcode and UMI will not have its own line in the bams.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T18:50:48.610Z
Answers.Text: @swbarnes2 There are indeed many tags, but isn't the UMI tag different than UMI counts? So how would I derive the expression matrix from these said tags? Thank you btw for your answer!
Answers.UserDisplayName: h3ab74
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T20:18:22.503Z
Answers.Text: Each read has its UMI as a bam tag.  You can go through the bam line by line and figure out which reads were assigned to what cell barcode, UMI and gene.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-27T14:00:45.450Z
Answers.Text: This reverse process is also complicated by the fact that error-correction is performed on the barcodes and UMIs as well. For scRNA-seq, you typically start with 4 FASTQs: UMIs, barcodes, R1, and R2. There will likely be errors in those sequences, so barcodes/UMIs are corrected for by comparing to a whitelist of known input barcodes. You can take a look at the "Feature Barcoding Tags" section of the 10x support page @swbarnes2 listed to see what tags to look for. As mentioned, it's not a simple BAM -> FASTQ operation, but you should be able to do it
Answers.UserDisplayName: James Hawley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-27T16:32:28.070Z
Answers.Text: @James, that's not at all how the fastqs come out for 10xGenomics samples. (Or dropSeq either)  And feature barcoding is something new and extra; an old experiment probably won't have it.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-30T14:00:30.630Z
Answers.Text: My mistake, I was mixing up the scATAC-seq and scRNA-seq protocols. The scATAC protocol will have 3 or 4 FASTQs (https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/using/fastq-input), whereas scRNA has 2 or 3 (https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/using/fastq-input). They both do barcode correction, but the input FASTQs are slightly different, and you're correct that the UMIs are probably only on newer datasets
Answers.UserDisplayName: James Hawley
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-30T17:38:19.910Z
Answers.Text: @James UMIs have been used from at least cellranger 1.1.0 in 2016, if not earlier. Features are different, and were just introduced this year.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T02:05:31.537Z
Answers.Text: Cell Ranger bam files have the corrected and uncorrected barcodes and UMIs.  It is definitely possible to go from a Cell Ranger bam file to fastqs. 10X has a tool for doing it (https://support.10xgenomics.com/docs/bamtofastq)
Answers.UserDisplayName: GWW
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T21:45:06.520Z
Answers.LastActivityDate: 2019-02-28T07:10:50.190Z
Answers.LastEditDate: 2019-02-28T07:10:50.190Z
Answers.UserDisplayName: swbarnes2
Answers.Body: <h3>How to re-analyze 10X BAM files?</h3>
  <p>This is a great question and honestly, I don't think there was an easy way to do this at the time the question was asked. The reason is that if you want to re-do the authors' analysis to get the gene expression (gene-cell / feature-barcode) matrix and they used <code>cellranger</code> (the official 10X pipeline software) to do that, you will need FASTQ files as input - because &quot;10x pipelines require sequencer FASTQs (with embedded barcodes) as input.&quot;</p>
  <p><strong>Therefore, to get FASTQ files from 10X BAM files, there's a relatively new tool provided by 10X Genomics: <a href="https://support.10xgenomics.com/docs/bamtofastq" rel="noreferrer"><code>bamtofastq</code></a>. Then you can simply use those FASTQ files as input to <code>cellranger</code>.</strong></p>
  <p>More info here: <a href="https://support.10xgenomics.com/docs/bamtofastq" rel="noreferrer">https://support.10xgenomics.com/docs/bamtofastq</a></p>
  <h3>How to correctly download 10X BAM files from NCBI?</h3>
  <p>The only caveat is that you need a BAM file generated directly by 10X's <code>cellranger</code> (or the respective 10X pipeline, if not dealing with gene expression) - that means that a BAM file obtained by downloading an SRA from NCBI and converting to BAM won't work; you need to get the original BAM file directly (often found among the originally submitted files, under &quot;Original format&quot;).</p>
  <p>To give you some additional help here (because figuring this out is also not trivial): an original 10X BAM file (if originally submitted in this format) can be downloaded from NCBI using <code>prefetch</code> from <a href="https://github.com/ncbi/sra-tools" rel="noreferrer"><code>sra-tools</code></a>, specifying the <code>--type</code> option - for example:</p>
  <pre class="lang-bash prettyprint-override"><code>prefetch --type TenX --max-size 100000000 SRR5167880
  </code></pre>
  <p>Because the file was larger than the default limit of 20GB, I also had to increase it by specifying the <code>--max-size</code> option.</p>
  <p>Alternatively, you can follow the official 10X <a href="https://support.10xgenomics.com/permalink/2CBLaOKISoo2m0eYAkcsaC" rel="noreferrer">Instructions to Download and Process BAM files of 1.3 Million Brain Cells</a>.</p>
  <p><em>I know this is an old thread but I'm hoping my answer will be helpful to anyone who has the same question (like me).</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-12T15:14:46.547Z
Answers.Text: This is the simplest approach (if the process’s data is not available on GEO). Computing the UMI count matrix for 10x Genomics data is already implemented in Cell Ranger and if the same version is used it should reproduce the results of the study.
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-10T17:49:14.883Z
Answers.LastActivityDate: 2021-03-10T17:49:14.883Z
Answers.UserDisplayName: Brunox13
Body: <p>I am trying to reproduce some results of a scRNASeq experiment. However I am new to the server-side aspect of such analyses and am very confused at the moment.</p>
  <p>The data provided by the authors of the paper is in .BAM format and from there I wish to derive a gene expression matrix (UMI counts per gene per cell). The authors stated that they did this using the 10X genomics pipeline, however it seems that .BAM outputs are sort of the &quot;final version&quot; of the pipeline and, the pipeline only really deals with truly raw sequencing files (bcl/fastq). I am trying to learn how to do all this processing myself as I will need to eventually. Unfortunately, most of the documents I have read end up making me more confused. I know BAM files are technically already processed and aligned but, how would I get the counts from the file?</p>
  <p>I have considered converting it back to <code>fastq</code> format and following the pipeline from the beginning, but it just seems like I would be moving backwards by doing such.</p>
  <p>I have also found some R packages that can integrate BAM files in the environment like <code>Rsamtools</code>, but I would much rather do this on the server and then upload a csv into R for the downstream analysis as the files are +60GB each.</p>
  <p>Essentially I am just asking for some advice, or a vignette/article I can read that would clear up how to proceed with the BAM outputs to derive the gene expression data. I have already read the <code>cellranger</code> pages on the 10X genomics website but they were of no help in answering this quesion.</p>
  <p>Thank you for your time,</p>
  
Tags: <sam><10x-genomics><cellranger><bash><umi>
Title: BAM to gene expression matrix (UMI counts per gene per cell),10X
ThreadText: # BAM to gene expression matrix (UMI counts per gene per cell),10X
  
  (Asked by: username_0 on 2019-02-21)
  
  I am trying to reproduce some results of a scRNASeq experiment. However I am new to the server-side aspect of such analyses and am very confused at the moment.
  
  The data provided by the authors of the paper is in .BAM format and from there I wish to derive a gene expression matrix (UMI counts per gene per cell). The authors stated that they did this using the 10X genomics pipeline, however it seems that .BAM outputs are sort of the "final version" of the pipeline and, the pipeline only really deals with truly raw sequencing files (bcl/fastq). I am trying to learn how to do all this processing myself as I will need to eventually. Unfortunately, most of the documents I have read end up making me more confused. I know BAM files are technically already processed and aligned but, how would I get the counts from the file?
  
  I have considered converting it back to `fastq` format and following the pipeline from the beginning, but it just seems like I would be moving backwards by doing such.
  
  I have also found some R packages that can integrate BAM files in the environment like `Rsamtools`, but I would much rather do this on the server and then upload a csv into R for the downstream analysis as the files are +60GB each.
  
  Essentially I am just asking for some advice, or a vignette/article I can read that would clear up how to proceed with the BAM outputs to derive the gene expression data. I have already read the `cellranger` pages on the 10X genomics website but they were of no help in answering this quesion.
  
  Thank you for your time,
  
  **Tags:** sam, 10x-genomics, cellranger, bash, umi
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2021-03-10. Score: 6
  
  ### How to re-analyze 10X BAM files?
  
  This is a great question and honestly, I don't think there was an easy way to do this at the time the question was asked. The reason is that if you want to re-do the authors' analysis to get the gene expression (gene-cell / feature-barcode) matrix and they used `cellranger` (the official 10X pipeline software) to do that, you will need FASTQ files as input - because "10x pipelines require sequencer FASTQs (with embedded barcodes) as input."
  
  **Therefore, to get FASTQ files from 10X BAM files, there's a relatively new tool provided by 10X Genomics: [`bamtofastq`](https://support.10xgenomics.com/docs/bamtofastq). Then you can simply use those FASTQ files as input to `cellranger`.**
  
  More info here: <https://support.10xgenomics.com/docs/bamtofastq>
  
  ### How to correctly download 10X BAM files from NCBI?
  
  The only caveat is that you need a BAM file generated directly by 10X's `cellranger` (or the respective 10X pipeline, if not dealing with gene expression) - that means that a BAM file obtained by downloading an SRA from NCBI and converting to BAM won't work; you need to get the original BAM file directly (often found among the originally submitted files, under "Original format").
  
  To give you some additional help here (because figuring this out is also not trivial): an original 10X BAM file (if originally submitted in this format) can be downloaded from NCBI using `prefetch` from [`sra-tools`](https://github.com/ncbi/sra-tools), specifying the `--type` option - for example:
  
  ```bash
  prefetch --type TenX --max-size 100000000 SRR5167880
  
  ```
  
  Because the file was larger than the default limit of 20GB, I also had to increase it by specifying the `--max-size` option.
  
  Alternatively, you can follow the official 10X [Instructions to Download and Process BAM files of 1.3 Million Brain Cells](https://support.10xgenomics.com/permalink/2CBLaOKISoo2m0eYAkcsaC).
  
  *I know this is an old thread but I'm hoping my answer will be helpful to anyone who has the same question (like me).*
  
  > **Comments:**
  > **username_5**: This is the simplest approach (if the process’s data is not available on GEO). Computing the UMI count matrix for 10x Genomics data is already implemented in Cell Ranger and if the same version is used it should reproduce the results of the study.  
  
  ---
  
  ### Answer by username_1 on 2019-02-21. Score: 4
  
  You should be able to parse out what you need using the tags in the .bam. 10xGenomics' website says what tags they add.
  
  <https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/bam>
  
  Going backwards would also involve parsing the tags, because you have to make two fastq files, and a simple bam -> fastq pipeline won't do that correctly.
  
  > **Comments:**
  > **username_0**: Could you be more specific please? I'm very new to this procedure and bash scripting in general.  
  > **username_1**: Yes, 10xGenomics single cell bams have the genes called in the bam tags.  https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/output/bam And no, samtools fastq will not make paired end fastqs suitable to rerun through cellranger, because the read data from the read containing the cell barcode and UMI will not have its own line in the bams.  
  > **username_0**: @username_1 There are indeed many tags, but isn't the UMI tag different than UMI counts? So how would I derive the expression matrix from these said tags? Thank you btw for your answer!  
  > **username_1**: Each read has its UMI as a bam tag.  You can go through the bam line by line and figure out which reads were assigned to what cell barcode, UMI and gene.  
  > **username_2**: This reverse process is also complicated by the fact that error-correction is performed on the barcodes and UMIs as well. For scRNA-seq, you typically start with 4 FASTQs: UMIs, barcodes, R1, and R2. There will likely be errors in those sequences, so barcodes/UMIs are corrected for by comparing to a whitelist of known input barcodes. You can take a look at the "Feature Barcoding Tags" section of the 10x support page @username_1 listed to see what tags to look for. As mentioned, it's not a simple BAM -> FASTQ operation, but you should be able to do it  
  > **username_1**: @username_2, that's not at all how the fastqs come out for 10xGenomics samples. (Or dropSeq either)  And feature barcoding is something new and extra; an old experiment probably won't have it.  
  > **username_2**: My mistake, I was mixing up the scATAC-seq and scRNA-seq protocols. The scATAC protocol will have 3 or 4 FASTQs (https://support.10xgenomics.com/single-cell-atac/software/pipelines/latest/using/fastq-input), whereas scRNA has 2 or 3 (https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/using/fastq-input). They both do barcode correction, but the input FASTQs are slightly different, and you're correct that the UMIs are probably only on newer datasets  
  > **username_1**: @username_2 UMIs have been used from at least cellranger 1.1.0 in 2016, if not earlier. Features are different, and were just introduced this year.  
  > **username_3**: Cell Ranger bam files have the corrected and uncorrected barcodes and UMIs.  It is definitely possible to go from a Cell Ranger bam file to fastqs. 10X has a tool for doing it (https://support.10xgenomics.com/docs/bamtofastq)  
  
  ---
  
  