Answers.Body: <p>My main question is whether you refer to the fastq files after adapter trimming or before. It seems you mean the original fastq files. In that case, any read filtering done by fastp will remove low quality or totally-adapter reads.</p>
  <p>Note from the <a href="https://github.com/OpenGene/fastp#filtering" rel="nofollow noreferrer">fastp docs</a> that length trimming is enabled by default, so trimmed reads will be removed if they are shorter than the length setting, even if they are otherwise high quality. And if they are low quality then they will be filtered out for sure. I refer you to the docs to see what all the various default settings are. Possibly you could turn them all off and still see the initial number of reads, but I very much doubt that they would be useful for you.</p>
  <p>UPDATE: see also @Yossi Farjoun's excellent point about alignments vs. reads- not at all the same thing! though you can learn the exact number of reads from a BAM (using e.g. <code>samtools stats</code>), the BAM is in fact a record of alignments. So it's important to be looking at the right number when you're doing your comparison!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-11T00:03:06.453Z
Answers.LastActivityDate: 2023-12-15T17:45:34.750Z
Answers.LastEditDate: 2023-12-15T17:45:34.750Z
Answers.UserDisplayName: Maximilian Press
Answers.Body: <p>&quot;Total number of reads in R1 and R2 FastQ files = mapped and unmapped reads in the corresponding BAM file.&quot;</p>
  <p>there might be confusion regarding the word &quot;read&quot;.</p>
  <p>&quot;record&quot; refers to an entry in the BAM file (a row in SAM)
  &quot;template&quot; refers to the original sequence from the fastq/sequencer.</p>
  <p>after mapping a single template may be the source of two different <em>records</em> due to secondary and supplementary alignments.</p>
  <p>So the general answer to your question is: Not if you count &quot;reads&quot; as rows in the SAM file Yes, if you count carefully and make sure to not double-count records that have the same &quot;query name&quot; (the name from the fastq file)</p>
  <p>For more information about SAM/BAM/CRAM the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="nofollow noreferrer">spec</a> can provide hours of light reading... :-)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-15T01:19:19.297Z
Answers.LastActivityDate: 2023-12-15T01:19:19.297Z
Answers.UserDisplayName: Yossi Farjoun
Body: <p>In a FastQ to BAM pipeline where only adapter trimming is performed, I've noticed a potential discrepancy in read counts between the initial FastQ files and their resulting BAM file. Specifically, I'm seeking clarification on whether the following statement holds true:</p>
  <p>&quot;Total number of reads in R1 and R2 FastQ files = mapped and unmapped reads in the corresponding BAM file.&quot;</p>
  <p>I understand that adapter trimming is the sole preprocessing step in this pipeline. However, I'm uncertain about the impact of factors such as PCR duplicates, mapping efficiency, and any potential loss during processing. The BAM file does contain PCR duplicates, and I'm wondering if these factors contribute to the observed differences in read counts.</p>
  <p>Can someone shed light on the expected relationship between the read counts in the FastQ files and the BAM file under these conditions? Any insights into how PCR duplicates may affect the discrepancy would be particularly helpful. Additionally, if there are common reasons for the observed differences in read counts between FastQ and BAM files, I'd appreciate guidance on identifying and addressing them.</p>
  <p><strong>Additional information:</strong></p>
  <p>The pipeline used:</p>
  <p>pre-trimming: <code>fastqc R1.fastq.gz R2.fastq.gz -o pre_trimming</code></p>
  <p>adapter trimming: <code>fastp -i R1.fastq.gz -o trimmed_R1.fastq.gz -I R2.fastq.gz -O trimmed_R2.fastq.gz -h file.html -j file.json</code></p>
  <p>alignment: <code>bwa mem -M -t 4 -R “@RG\tID:FE_S211\tPL:ILLUMINA\tSM:A4” hg19.fa trimmed_R1.fastq.gz trimmed_R2.fastq.gz | samtools sort -o output.bam -O BAM -@ 4 -</code></p>
  
Tags: <sam><fastq>
Title: Discrepancy in Read Counts Between FastQ and BAM Files in Adapter-Trimmed Pipeline
ThreadText: # Discrepancy in Read Counts Between FastQ and BAM Files in Adapter-Trimmed Pipeline
  
  (Asked by: username_0 on 2023-12-07)
  
  In a FastQ to BAM pipeline where only adapter trimming is performed, I've noticed a potential discrepancy in read counts between the initial FastQ files and their resulting BAM file. Specifically, I'm seeking clarification on whether the following statement holds true:
  
  "Total number of reads in R1 and R2 FastQ files = mapped and unmapped reads in the corresponding BAM file."
  
  I understand that adapter trimming is the sole preprocessing step in this pipeline. However, I'm uncertain about the impact of factors such as PCR duplicates, mapping efficiency, and any potential loss during processing. The BAM file does contain PCR duplicates, and I'm wondering if these factors contribute to the observed differences in read counts.
  
  Can someone shed light on the expected relationship between the read counts in the FastQ files and the BAM file under these conditions? Any insights into how PCR duplicates may affect the discrepancy would be particularly helpful. Additionally, if there are common reasons for the observed differences in read counts between FastQ and BAM files, I'd appreciate guidance on identifying and addressing them.
  
  **Additional information:**
  
  The pipeline used:
  
  pre-trimming: `fastqc R1.fastq.gz R2.fastq.gz -o pre_trimming`
  
  adapter trimming: `fastp -i R1.fastq.gz -o trimmed_R1.fastq.gz -I R2.fastq.gz -O trimmed_R2.fastq.gz -h file.html -j file.json`
  
  alignment: `bwa mem -M -t 4 -R “@RG\tID:FE_S211\tPL:ILLUMINA\tSM:A4” hg19.fa trimmed_R1.fastq.gz trimmed_R2.fastq.gz | samtools sort -o output.bam -O BAM -@ 4 -`
  
  **Tags:** sam, fastq
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-12-11. Score: 2
  
  My main question is whether you refer to the fastq files after adapter trimming or before. It seems you mean the original fastq files. In that case, any read filtering done by fastp will remove low quality or totally-adapter reads.
  
  Note from the [fastp docs](https://github.com/OpenGene/fastp#filtering) that length trimming is enabled by default, so trimmed reads will be removed if they are shorter than the length setting, even if they are otherwise high quality. And if they are low quality then they will be filtered out for sure. I refer you to the docs to see what all the various default settings are. Possibly you could turn them all off and still see the initial number of reads, but I very much doubt that they would be useful for you.
  
  UPDATE: see also @username_2's excellent point about alignments vs. reads- not at all the same thing! though you can learn the exact number of reads from a BAM (using e.g. `samtools stats`), the BAM is in fact a record of alignments. So it's important to be looking at the right number when you're doing your comparison!
  
  
  ---
  
  ### Answer by username_2 on 2023-12-15. Score: 2
  
  "Total number of reads in R1 and R2 FastQ files = mapped and unmapped reads in the corresponding BAM file."
  
  there might be confusion regarding the word "read".
  
  "record" refers to an entry in the BAM file (a row in SAM)
  "template" refers to the original sequence from the fastq/sequencer.
  
  after mapping a single template may be the source of two different *records* due to secondary and supplementary alignments.
  
  So the general answer to your question is: Not if you count "reads" as rows in the SAM file Yes, if you count carefully and make sure to not double-count records that have the same "query name" (the name from the fastq file)
  
  For more information about SAM/BAM/CRAM the [spec](https://samtools.github.io/hts-specs/SAMv1.pdf) can provide hours of light reading... :-)
  
  
  ---
  
  