Answers.Body: <p>The first error seems to come from picard when sorting the sam file:</p>
  <blockquote>
  <p>SAMFormatException: SAM validation error: ERROR: Read name M02699:57:000000000-LG9CK:1:1119:16584:12259, CIGAR M operator maps off end of reference</p>
  </blockquote>
  <p>From <a href="http://broadinstitute.github.io/picard/faq.html" rel="nofollow noreferrer">picard FAQ</a>:</p>
  <blockquote>
  <p>Picard validation errors may be turned into warnings by passing the command line argument VALIDATION_STRINGENCY=LENIENT.
  Picard validation messages may be suppressed completely with VALIDATION_STRINGENCY=SILENT.
  Another option is to use the CleanSam tool to soft-clip these reads so they don't map off the end of the reference.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-16T20:33:40.290Z
Answers.LastActivityDate: 2024-05-16T20:33:40.290Z
Answers.UserDisplayName: Cloudberry
Body: <p>I am trying to map MiSeq reads to a reference genome and extract mutations using MToolBox, which implements gsnap, GATK, Picard, and other tools. When running the tool with example data, there were no errors, so I believe the tool is installed correctly. However, when I used the data sequenced with MiSeq for this run, I encountered the following error. Specifically, the OUTPUT file (OUT.sam.bam) from SORTING OUT.sam FILES WITH PICARDTOOLS was not generated. How should I address this issue? Could it be a problem with the sequence data?</p>
  <p>Could someone please help me?</p>
  <pre><code>##### EXECUTING READ MAPPING WITH MAPEXOME...
  
  
  mapExome for sample pc-mtGXL, files found: pc-mtGXL.R1.fastq.gz pc-mtGXL.R2.fastq.gz
  Mapping onto mtDNA... /lustre7/home/User/MToolBox-1.2.1/MToolBox/bin/gmap/bin/gsnap -D /lustre7/home/User/MToolBox-1.2.1/MToolBox/gmapdb/ --gunzip -d chrM -A sam --nofails --pairmax-dna=500 --query-unk-mismatch=1
  --read-group-id=sample --read-group-name=sample --read-group-library=sample --read-group-platform=sample -n 1 -Q -O -t 8 pc-mtGXL.R1.fastq.gz pc-mtGXL.R2.fastq.gz &gt; /home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/outmt.sam 2&gt; /home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/logmt.txt Extracting FASTQ from SAM... Mapping onto complete human genome...single reads Mapping onto complete human genome...pair reads Reading Results... Filtering reads... Outfile saved on /home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/OUT.sam. Done.
  
  
  SAM files post-processing...
  
  ##### SORTING OUT.sam FILES WITH PICARDTOOLS...
  
  [Thu May 16 12:23:08 JST 2024] net.sf.picard.sam.SortSam INPUT=OUT.sam OUTPUT=OUT.sam.bam SORT_ORDER=coordinate TMP_DIR=[/home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/tmp]    VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false [Thu May 16 12:23:08 JST 2024] Executing as User@at138 on Linux 5.15.0-87-generic amd64; OpenJDK 64-Bit Server VM
  1.8.0_392-b08; Picard version: 1.98(1547) [Thu May 16 12:23:08 JST 2024] net.sf.picard.sam.SortSam done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=2058354688 To get help, see http://picard.sourceforge.net/index.shtml#GettingHelp Exception in thread &quot;main&quot; net.sf.samtools.SAMFormatException: SAM validation error: ERROR: Read name M02699:57:000000000-LG9CK:1:1119:16584:12259, CIGAR M operator maps off end of reference     at net.sf.samtools.SAMUtils.processValidationErrors(SAMUtils.java:448)  at net.sf.samtools.SAMRecord.getCigar(SAMRecord.java:606)   at net.sf.samtools.SAMRecord.getCigarLength(SAMRecord.java:617)     at net.sf.samtools.SAMRecord.isValid(SAMRecord.java:1599)   at net.sf.samtools.SAMLineParser.parseLine(SAMLineParser.java:328)  at net.sf.samtools.SAMTextReader<span class="math-container">$RecordIterator.parseLine(SAMTextReader.java:237)   at net.sf.samtools.SAMTextReader$</span>RecordIterator.next(SAMTextReader.java:225)    at net.sf.samtools.SAMTextReader<span class="math-container">$RecordIterator.next(SAMTextReader.java:201)    at net.sf.samtools.SAMFileReader$</span>AssertableIterator.next(SAMFileReader.java:672)    at net.sf.samtools.SAMFileReader$AssertableIterator.next(SAMFileReader.java:650)    at net.sf.picard.sam.SortSam.doWork(SortSam.java:68)    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)   at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:119)   at net.sf.picard.sam.SortSam.main(SortSam.java:57) Success.
  
  samtools index: &quot;OUT.sam.bam&quot; is in a format that cannot be usefully indexed
  
  ##### REALIGNING KNOWN INDELS WITH GATK INDELREALIGNER...
  
  Realigning known indels for file OUT_pc-mtGXL/OUT.sam.bam using /home/User/MToolBox-1.2.1/MToolBox/data/MITOMAP_HMTDB_known_indels.chrM as reference... INFO  12:23:11,978 HelpFormatter -
  ------------------------------------------------------------------------------------  INFO  12:23:11,984 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-1-0-gf15c1c3ef, Compiled 2018/02/19 05:43:50  INFO  12:23:11,984 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute  INFO  12:23:11,984 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk  INFO  12:23:11,984 HelpFormatter - [Thu May 16 12:23:11 JST 2024] Executing on Linux
  5.15.0-87-generic amd64  INFO  12:23:11,984 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_392-b08  INFO  12:23:11,986 HelpFormatter - Program Args: -U ALLOW_N_CIGAR_READS -T IndelRealigner -R /home/User/MToolBox-1.2.1/MToolBox//data/chrM.fa -I OUT.sam.bam -o OUT.realigned.bam -targetIntervals /home/User/MToolBox-1.2.1/MToolBox//data/intervals_file_chrM.list
  -known /home/User/MToolBox-1.2.1/MToolBox//data/MITOMAP_HMTDB_known_indels_chrM.vcf
  -compress 0  INFO  12:23:11,993 HelpFormatter - Executing as User@at138 on Linux 5.15.0-87-generic amd64; OpenJDK 64-Bit Server VM
  1.8.0_392-b08.  INFO  12:23:11,994 HelpFormatter - Date/Time: 2024/05/16 12:23:11  INFO  12:23:11,994 HelpFormatter -
  ------------------------------------------------------------------------------------  INFO  12:23:11,994 HelpFormatter -
  ------------------------------------------------------------------------------------  INFO  12:23:12,139 NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/lustre7/home/User/MToolBox-1.2.1/MToolBox/ext_tools/GenomeAnalysisTK.jar!/com/intel/gkl/native/libgkl_compression.so INFO  12:23:12,711 GenomeAnalysisEngine - Deflater: IntelDeflater  INFO  12:23:12,711 GenomeAnalysisEngine - Inflater: IntelInflater  INFO  12:23:12,711 GenomeAnalysisEngine - Strictness is SILENT  INFO  12:23:12,962 GenomeAnalysisEngine - Downsampling Settings: No downsampling  INFO  12:23:12,967 SAMDataSource<span class="math-container">$SAMReaders - Initializing SAMRecords in serial  INFO  12:23:12,985 SAMDataSource$</span>SAMReaders - Done initializing BAM readers: total time
  0.02 
  ##### ERROR ------------------------------------------------------------------------------------------
  ##### ERROR A USER ERROR has occurred (version 3.8-1-0-gf15c1c3ef): 
  ##### ERROR
  ##### ERROR This means that one or more arguments or inputs in your command are incorrect.
  ##### ERROR The error message below tells you what is the problem.
  ##### ERROR
  ##### ERROR If the problem is an invalid argument, please check the online documentation guide
  ##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
  ##### ERROR
  ##### ERROR Visit our website and forum for extensive documentation and answers to 
  ##### ERROR commonly asked questions &gt;https://software.broadinstitute.org/gatk
  ##### ERROR
  ##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
  ##### ERROR
  ##### ERROR MESSAGE: SAM/BAM/CRAM file OUT.sam.bam is malformed. &gt;Please see https://software.broadinstitute.org/gatk/documentation/article?id=1317for more information. Error details: SAM file doesn't have any read groups defined in the header.  The GATK no longer supports SAM files without read groups
  ##### ERROR ------------------------------------------------------------------------------------------
  
  The last process reported an error. Exit.
  </code></pre>
  <p><code>OUT.sam.bam</code> is 0 KB (i.e. empty). It is human mtDNA.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-17T11:11:56.107Z
Comments.Text: Seems that `MToolBox` is quite frozen in time (last release (Sep 2019) with hg19 and recommended tools like GSNAP 2015-12-31.v7 and samtools 1.3. Are there any alternative packages/pipelines being maintained?
Comments.UserDisplayName: darked89
Tags: <sam><samtools><gatk><picard><mtdna>
Title: Problem while mapping reads to mtDNA (SortSam)
ThreadText: # Problem while mapping reads to mtDNA (SortSam)
  
  (Asked by: username_0 on 2024-05-16)
  
  I am trying to map MiSeq reads to a reference genome and extract mutations using MToolBox, which implements gsnap, GATK, Picard, and other tools. When running the tool with example data, there were no errors, so I believe the tool is installed correctly. However, when I used the data sequenced with MiSeq for this run, I encountered the following error. Specifically, the OUTPUT file (OUT.sam.bam) from SORTING OUT.sam FILES WITH PICARDTOOLS was not generated. How should I address this issue? Could it be a problem with the sequence data?
  
  Could someone please help me?
  
  ```
  ##### EXECUTING READ MAPPING WITH MAPEXOME...
  
  mapExome for sample pc-mtGXL, files found: pc-mtGXL.R1.fastq.gz pc-mtGXL.R2.fastq.gz
  Mapping onto mtDNA... /lustre7/home/User/MToolBox-1.2.1/MToolBox/bin/gmap/bin/gsnap -D /lustre7/home/User/MToolBox-1.2.1/MToolBox/gmapdb/ --gunzip -d chrM -A sam --nofails --pairmax-dna=500 --query-unk-mismatch=1
  --read-group-id=sample --read-group-name=sample --read-group-library=sample --read-group-platform=sample -n 1 -Q -O -t 8 pc-mtGXL.R1.fastq.gz pc-mtGXL.R2.fastq.gz > /home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/outmt.sam 2> /home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/logmt.txt Extracting FASTQ from SAM... Mapping onto complete human genome...single reads Mapping onto complete human genome...pair reads Reading Results... Filtering reads... Outfile saved on /home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/OUT.sam. Done.
  
  SAM files post-processing...
  
  ##### SORTING OUT.sam FILES WITH PICARDTOOLS...
  
  [Thu May 16 12:23:08 JST 2024] net.sf.picard.sam.SortSam INPUT=OUT.sam OUTPUT=OUT.sam.bam SORT_ORDER=coordinate TMP_DIR=[/home/User/MTDNA-prospective/pc_GXL/out/OUT_pc-mtGXL/tmp]    VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false [Thu May 16 12:23:08 JST 2024] Executing as User@at138 on Linux 5.15.0-87-generic amd64; OpenJDK 64-Bit Server VM
  1.8.0_392-b08; Picard version: 1.98(1547) [Thu May 16 12:23:08 JST 2024] net.sf.picard.sam.SortSam done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=2058354688 To get help, see http://picard.sourceforge.net/index.shtml#GettingHelp Exception in thread "main" net.sf.samtools.SAMFormatException: SAM validation error: ERROR: Read name M02699:57:000000000-LG9CK:1:1119:16584:12259, CIGAR M operator maps off end of reference     at net.sf.samtools.SAMUtils.processValidationErrors(SAMUtils.java:448)  at net.sf.samtools.SAMRecord.getCigar(SAMRecord.java:606)   at net.sf.samtools.SAMRecord.getCigarLength(SAMRecord.java:617)     at net.sf.samtools.SAMRecord.isValid(SAMRecord.java:1599)   at net.sf.samtools.SAMLineParser.parseLine(SAMLineParser.java:328)  at net.sf.samtools.SAMTextReader$RecordIterator.parseLine(SAMTextReader.java:237)   at net.sf.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:225)    at net.sf.samtools.SAMTextReader$RecordIterator.next(SAMTextReader.java:201)    at net.sf.samtools.SAMFileReader$AssertableIterator.next(SAMFileReader.java:672)    at net.sf.samtools.SAMFileReader$AssertableIterator.next(SAMFileReader.java:650)    at net.sf.picard.sam.SortSam.doWork(SortSam.java:68)    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)   at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:119)   at net.sf.picard.sam.SortSam.main(SortSam.java:57) Success.
  
  samtools index: "OUT.sam.bam" is in a format that cannot be usefully indexed
  
  ##### REALIGNING KNOWN INDELS WITH GATK INDELREALIGNER...
  
  Realigning known indels for file OUT_pc-mtGXL/OUT.sam.bam using /home/User/MToolBox-1.2.1/MToolBox/data/MITOMAP_HMTDB_known_indels.chrM as reference... INFO  12:23:11,978 HelpFormatter -
  ------------------------------------------------------------------------------------  INFO  12:23:11,984 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.8-1-0-gf15c1c3ef, Compiled 2018/02/19 05:43:50  INFO  12:23:11,984 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute  INFO  12:23:11,984 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk  INFO  12:23:11,984 HelpFormatter - [Thu May 16 12:23:11 JST 2024] Executing on Linux
  5.15.0-87-generic amd64  INFO  12:23:11,984 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_392-b08  INFO  12:23:11,986 HelpFormatter - Program Args: -U ALLOW_N_CIGAR_READS -T IndelRealigner -R /home/User/MToolBox-1.2.1/MToolBox//data/chrM.fa -I OUT.sam.bam -o OUT.realigned.bam -targetIntervals /home/User/MToolBox-1.2.1/MToolBox//data/intervals_file_chrM.list
  -known /home/User/MToolBox-1.2.1/MToolBox//data/MITOMAP_HMTDB_known_indels_chrM.vcf
  -compress 0  INFO  12:23:11,993 HelpFormatter - Executing as User@at138 on Linux 5.15.0-87-generic amd64; OpenJDK 64-Bit Server VM
  1.8.0_392-b08.  INFO  12:23:11,994 HelpFormatter - Date/Time: 2024/05/16 12:23:11  INFO  12:23:11,994 HelpFormatter -
  ------------------------------------------------------------------------------------  INFO  12:23:11,994 HelpFormatter -
  ------------------------------------------------------------------------------------  INFO  12:23:12,139 NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/lustre7/home/User/MToolBox-1.2.1/MToolBox/ext_tools/GenomeAnalysisTK.jar!/com/intel/gkl/native/libgkl_compression.so INFO  12:23:12,711 GenomeAnalysisEngine - Deflater: IntelDeflater  INFO  12:23:12,711 GenomeAnalysisEngine - Inflater: IntelInflater  INFO  12:23:12,711 GenomeAnalysisEngine - Strictness is SILENT  INFO  12:23:12,962 GenomeAnalysisEngine - Downsampling Settings: No downsampling  INFO  12:23:12,967 SAMDataSource$SAMReaders - Initializing SAMRecords in serial  INFO  12:23:12,985 SAMDataSource$SAMReaders - Done initializing BAM readers: total time
  0.02 
  ##### ERROR ------------------------------------------------------------------------------------------
  ##### ERROR A USER ERROR has occurred (version 3.8-1-0-gf15c1c3ef): 
  ##### ERROR
  ##### ERROR This means that one or more arguments or inputs in your command are incorrect.
  ##### ERROR The error message below tells you what is the problem.
  ##### ERROR
  ##### ERROR If the problem is an invalid argument, please check the online documentation guide
  ##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
  ##### ERROR
  ##### ERROR Visit our website and forum for extensive documentation and answers to 
  ##### ERROR commonly asked questions >https://software.broadinstitute.org/gatk
  ##### ERROR
  ##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
  ##### ERROR
  ##### ERROR MESSAGE: SAM/BAM/CRAM file OUT.sam.bam is malformed. >Please see https://software.broadinstitute.org/gatk/documentation/article?id=1317for more information. Error details: SAM file doesn't have any read groups defined in the header.  The GATK no longer supports SAM files without read groups
  ##### ERROR ------------------------------------------------------------------------------------------
  
  The last process reported an error. Exit.
  
  ```
  
  `OUT.sam.bam` is 0 KB (i.e. empty). It is human mtDNA.
  
  **Tags:** sam, samtools, gatk, picard, mtdna
  
  > **Comments:**
  > **username_1**: Seems that `MToolBox` is quite frozen in time (last release (Sep 2019) with hg19 and recommended tools like GSNAP 2015-12-31.v7 and samtools 1.3. Are there any alternative packages/pipelines being maintained?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-05-16. Score: 1
  
  The first error seems to come from picard when sorting the sam file:
  
  > SAMFormatException: SAM validation error: ERROR: Read name M02699:57:000000000-LG9CK:1:1119:16584:12259, CIGAR M operator maps off end of reference
  
  From [picard FAQ](http://broadinstitute.github.io/picard/faq.html):
  
  > Picard validation errors may be turned into warnings by passing the command line argument VALIDATION\_STRINGENCY=LENIENT.
  > Picard validation messages may be suppressed completely with VALIDATION\_STRINGENCY=SILENT.
  > Another option is to use the CleanSam tool to soft-clip these reads so they don't map off the end of the reference.
  
  
  ---
  
  