Answers.Body: <p>The common solution for scRNA-seq is to put cell barcodes and such in read headers and then post-process things with UMItools.</p>
  
  <p>But regarding your actual question, STAR can accept SAM/BAM as input with the <code>--readFilesType SAM PE</code> option (it's <code>SAM</code> for both that format and BAM). You can swap <code>SE</code> for <code>PE</code> if your data is single-end (or is effectively that way, as is the case in many scRNA-seq experiments). Note that in the case of PE experiments that mates <a href="https://github.com/alexdobin/STAR/issues/416" rel="noreferrer">must be next to each other</a>. I really don't know what this does with auxiliary tags though, so you would need to see if they're lost before you start processing a lot of data).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-16T10:14:14.160Z
Answers.LastActivityDate: 2018-11-16T10:21:36.733Z
Answers.LastEditDate: 2018-11-16T10:21:36.733Z
Answers.UserDisplayName: Devon Ryan
Answers.Body: <p>With bwa-mem or minimap2, the recommended way is</p>
  
  <pre><code>samtools fastq -T BC,RX name-grouped.sam | bwa mem -C -p -
  </code></pre>
  
  <p>This passes the <code>BC</code> and <code>RX</code> tags through bwa-mem and copies them to the SAM output. If you are running paired-end alignment and your BAM is coordinate sorted, you have to run <code>samtools collate</code> to group reads by name:</p>
  
  <pre><code>samtools collate -Ol0 in.bam tmp | samtools fastq -T BC,RX - | bwa mem -pC -
  </code></pre>
  
  <p>Here <code>bwa mem -C</code> or <code>minimap2 -y</code> copies FASTQ comments to SAM output. This strategy is much easier to deal with than SAM/BAM. A caveat is that it doesn't work well for read groups because each read group requires a separate header line. You have to manually modify the header.</p>
  
  <p>I am not sure which short-read RNA-seq mappers have implemented a similar feature. You might consider to send a feature request.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-19T14:48:15.237Z
Answers.LastActivityDate: 2018-11-19T16:52:53.583Z
Answers.LastEditDate: 2018-11-19T16:52:53.583Z
Answers.UserDisplayName: user172818
Answers.Body: <p>At the Broad Institute, we do something similar to what is described by user172818. We use uBAM (unmapped BAM) to store unmapped reads, because as you say the SAM format has a more appropriate way to encode metadata. However, we use a Picard tool called SamToFastq to do the initial conversion then pipe the output of that to BWA mem. Then we use an additional step to reattach metadata, read group info and so on, using a Picard tool called MergeBamAlignments. </p>
  
  <p>You can see the full implementation here (starts as line 71): </p>
  
  <p><a href="https://github.com/gatk-workflows/gatk4-genome-processing-pipeline/blob/master/tasks/Alignment.wdl" rel="nofollow noreferrer">https://github.com/gatk-workflows/gatk4-genome-processing-pipeline/blob/master/tasks/Alignment.wdl</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T18:59:03.943Z
Answers.LastActivityDate: 2020-04-23T18:59:03.943Z
Answers.UserDisplayName: Geraldine_VdAuwera
Answers.Body: <p><a href="https://github.com/amplab/snap" rel="nofollow noreferrer">SNAP</a> is a hash-index based genomic aligner that can take SAM and BAM as input. It has been used as an aligner for RNA-seq (see <a href="https://hood-price.systemsbiology.org/research/snapr/" rel="nofollow noreferrer">SNAPR</a>) and BS-seq (see <a href="https://github.com/khuang28jhu/bs3" rel="nofollow noreferrer">BS-Seeker3</a>). SNAPR seems to accept SAM/BAM but it is not clear whether BS-Seeker3 wraps this functionality as well.</p>
  
  <p>I tried SNAP and it seems to do the job well, i.e. it only adds tags during alignment and does not remove metadata that has been deposited prior to alignment, neither from the SAM header nor from individual reads.</p>
  
  <p><strong>NOTE</strong> I post this answer to my own question after I became aware of SNAP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-19T14:45:50.560Z
Answers.LastActivityDate: 2018-11-19T16:24:29.050Z
Answers.LastEditDate: 2018-11-19T16:24:29.050Z
Answers.UserDisplayName: okartal
Body: <p>When we get raw sequenced reads in FASTQ format, we usually need to do some pre-processing (QC, demultiplexing, etc.) prior to sequence alignment. The only way to register the results of this preprocessing is to add metadata to each read. This is best done by using the SAM format which has a well-defined system of metadata tags.</p>
  
  <p>However, it seems that not a single read mapper (except maybe for <em>bwa aln</em> with option <em>-b</em>) accepts SAM (or one of the binary equivalents BAM/CRAM) as input to make use of the metadata that has been added during preprocessing. This implies that one has to convert the unaligned reads back to FASTQ format. </p>
  
  <p>In any case, whether one uses FASTQ throughout or converts back to FASTQ, a common annoyance with FASTQ is that metadata is registered as part of the query name - this is not standardized or even well-defined. After alignment, we again have to do some text processing to get the metadata out of the query name and into proper metadata tags defined by SAM. This back and forth hacking is highly unfortunate and error prone. I have seen this, for <strong>example</strong>, in some single-cell sequencing projects where we have highly multiplexed sequencing runs with reads that need to be QC'd and classified. The new SAM standard from May 2018 does indeed define proper tags for handling cell-multiplexed runs. SAM is the de-facto standard to store both unaligned and aligned reads for years to come so it should be standard for read mappers to accept it as input.</p>
  
  <p><strong>I would be grateful for some hints and if we could collect here some read mappers that do accept SAM/BAM/CRAM as input format.</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-15T09:25:09.900Z
Comments.Text: Why not store the read identifiers and metadata in a separate table file?
Comments.UserDisplayName: Chris_Rands
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-16T06:51:57.960Z
Comments.Text: Of course, this is possible. But apart from not being an answer to my question, this is not a good solution because it is not standardized and invites inconsistencies. You have to write special code to read the table. The SAM format (which is also a table) is maybe not the ideal but it is the best we have. If we could all stick to the SAM format instead of introducing custom formats the interoperability of bioinformatics tools would be much better.
Comments.UserDisplayName: okartal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-20T12:23:23.367Z
Comments.Text: I’m a bit confused because the preprocessing you mention is pretty routine, and is generally done without the involvement of BAM files.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-11T18:40:24.897Z
Comments.Text: Please remember to confirm an answer once you've received one.
Comments.UserDisplayName: Stephopolis
Tags: <sam><ngs><read-mapping><data-preprocessing>
Title: Is there a read mapper that allows input in SAM format instead of FASTA/FASTQ?
ThreadText: # Is there a read mapper that allows input in SAM format instead of FASTA/FASTQ?
  
  (Asked by: username_0 on 2018-11-15)
  
  When we get raw sequenced reads in FASTQ format, we usually need to do some pre-processing (QC, demultiplexing, etc.) prior to sequence alignment. The only way to register the results of this preprocessing is to add metadata to each read. This is best done by using the SAM format which has a well-defined system of metadata tags.
  
  However, it seems that not a single read mapper (except maybe for *bwa aln* with option *-b*) accepts SAM (or one of the binary equivalents BAM/CRAM) as input to make use of the metadata that has been added during preprocessing. This implies that one has to convert the unaligned reads back to FASTQ format.
  
  In any case, whether one uses FASTQ throughout or converts back to FASTQ, a common annoyance with FASTQ is that metadata is registered as part of the query name - this is not standardized or even well-defined. After alignment, we again have to do some text processing to get the metadata out of the query name and into proper metadata tags defined by SAM. This back and forth hacking is highly unfortunate and error prone. I have seen this, for **example**, in some single-cell sequencing projects where we have highly multiplexed sequencing runs with reads that need to be QC'd and classified. The new SAM standard from May 2018 does indeed define proper tags for handling cell-multiplexed runs. SAM is the de-facto standard to store both unaligned and aligned reads for years to come so it should be standard for read mappers to accept it as input.
  
  **I would be grateful for some hints and if we could collect here some read mappers that do accept SAM/BAM/CRAM as input format.**
  
  **Tags:** sam, ngs, read-mapping, data-preprocessing
  
  > **Comments:**
  > **username_1**: Why not store the read identifiers and metadata in a separate table file?  
  > **username_0**: Of course, this is possible. But apart from not being an answer to my question, this is not a good solution because it is not standardized and invites inconsistencies. You have to write special code to read the table. The SAM format (which is also a table) is maybe not the ideal but it is the best we have. If we could all stick to the SAM format instead of introducing custom formats the interoperability of bioinformatics tools would be much better.  
  > **username_2**: I’m a bit confused because the preprocessing you mention is pretty routine, and is generally done without the involvement of BAM files.  
  > **username_3**: Please remember to confirm an answer once you've received one.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2018-11-16. Score: 5
  
  The common solution for scRNA-seq is to put cell barcodes and such in read headers and then post-process things with UMItools.
  
  But regarding your actual question, STAR can accept SAM/BAM as input with the `--readFilesType SAM PE` option (it's `SAM` for both that format and BAM). You can swap `SE` for `PE` if your data is single-end (or is effectively that way, as is the case in many scRNA-seq experiments). Note that in the case of PE experiments that mates [must be next to each other](https://github.com/alexdobin/STAR/issues/416). I really don't know what this does with auxiliary tags though, so you would need to see if they're lost before you start processing a lot of data).
  
  
  ---
  
  ### Answer by username_5 on 2018-11-19. Score: 5
  
  With bwa-mem or minimap2, the recommended way is
  
  ```
  samtools fastq -T BC,RX name-grouped.sam | bwa mem -C -p -
  
  ```
  
  This passes the `BC` and `RX` tags through bwa-mem and copies them to the SAM output. If you are running paired-end alignment and your BAM is coordinate sorted, you have to run `samtools collate` to group reads by name:
  
  ```
  samtools collate -Ol0 in.bam tmp | samtools fastq -T BC,RX - | bwa mem -pC -
  
  ```
  
  Here `bwa mem -C` or `minimap2 -y` copies FASTQ comments to SAM output. This strategy is much easier to deal with than SAM/BAM. A caveat is that it doesn't work well for read groups because each read group requires a separate header line. You have to manually modify the header.
  
  I am not sure which short-read RNA-seq mappers have implemented a similar feature. You might consider to send a feature request.
  
  
  ---
  
  ### Answer by username_6 on 2020-04-23. Score: 1
  
  At the Broad Institute, we do something similar to what is described by username_5. We use uBAM (unmapped BAM) to store unmapped reads, because as you say the SAM format has a more appropriate way to encode metadata. However, we use a Picard tool called SamToFastq to do the initial conversion then pipe the output of that to BWA mem. Then we use an additional step to reattach metadata, read group info and so on, using a Picard tool called MergeBamAlignments.
  
  You can see the full implementation here (starts as line 71):
  
  <https://github.com/gatk-workflows/gatk4-genome-processing-pipeline/blob/master/tasks/Alignment.wdl>
  
  
  ---
  
  ### Answer by username_0 on 2018-11-19. Score: 0
  
  [SNAP](https://github.com/amplab/snap) is a hash-index based genomic aligner that can take SAM and BAM as input. It has been used as an aligner for RNA-seq (see [SNAPR](https://hood-price.systemsbiology.org/research/snapr/)) and BS-seq (see [BS-Seeker3](https://github.com/khuang28jhu/bs3)). SNAPR seems to accept SAM/BAM but it is not clear whether BS-Seeker3 wraps this functionality as well.
  
  I tried SNAP and it seems to do the job well, i.e. it only adds tags during alignment and does not remove metadata that has been deposited prior to alignment, neither from the SAM header nor from individual reads.
  
  **NOTE** I post this answer to my own question after I became aware of SNAP.
  
  
  ---
  
  