Answers.Body: <p>I'd advise you to keep away from ZUMIs unless you're doing the same workflow that it was designed for. I spent many days trying to sort out issues with that program, and eventually gave up; it wasn't created with the expectation of errors or unexpected input, and had too many programming minefields for the data I was working with.</p>
  <p>ZUMIs should be stopping on the first error and explaining what's wrong. As demonstrated by your situation, it's not; it's trying to keep on processing, barreling through and dragging errors along for the ride. It's not exactly beginner-friendly code.</p>
  <p>In the end, I changed to using Salmon/Alevin, which had a more fail-resistant interface, and (eventually) similar barcode flexibility:</p>
  <p><a href="https://combine-lab.github.io/alevin-fry-tutorials/2021/custom-geo/" rel="nofollow noreferrer">https://combine-lab.github.io/alevin-fry-tutorials/2021/custom-geo/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T07:56:54.003Z
Answers.Text: 0
  
  
  Thank you for your quick answer. I did the SmartSeq 3 protocol and it is advised to do bioinformatic analysis with zUMIs pipeline because of the specificities of results. Do you know is the Salmon/Alevin suitable for this protocol?
Answers.UserDisplayName: Isidora Pantovic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T10:33:43.343Z
Answers.Text: I don't know, sorry. If you've been advised to use ZUMIs, then do that, and try to get more help from the person who gave you that advice.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T06:03:44.193Z
Answers.LastActivityDate: 2023-08-24T19:35:34.050Z
Answers.LastEditDate: 2023-08-24T19:35:34.050Z
Answers.UserDisplayName: gringer
Body: <p>I wrote the .yaml file which was according the instruction from zUMIs:</p>
  <pre><code>###########################################
  #Welcome to zUMIs
  #below, please fill the mandatory inputs
  #We expect full paths for all files.
  ###########################################
  
  #define a project name that will be used to name output files
  project: SmartSeq3
  
  #Sequencing File Inputs:
  #For each input file, make one list object &amp; define path and barcode ranges 
  #base definition vocabulary: BC(n) UMI(n) cDNA(n).
  #Barcode range definition needs to account for all ranges.You can  give several comma-separated ranges for BC &amp; UMI sequences, eg.BC(1-6,20-26)
  #you can specify between 1 and 4 input files
  
  sequence_files:
    file1:
      name: /home/isidora/IRCCS_Candiolo/Fastq/*_R1_001_val_1.fq ##/home/isidora/IRCCS_Candiolo/Fastq/zUMIsFiles1/ #path to first file
      base_definition:
        - cDNA(1-150)
        - BC(1-8)
        - UMI(12-19)
    file2:
      name: /home/isidora/IRCCS_Candiolo/Fastq/*_R2_001_val_2.fq ## /home/isidora/IRCCS_Candiolo/Fastq/zUMIsFiles2/ #path to second file
      base_definition:
        - cDNA(1-150)
        - BC(1-8)
        - UMI(12-19)
  
  #reference genome setup
  reference:
    STAR_index: /home/isidora/IRCCS_Candiolo/Fastq/ #path to STAR genome index
    GTF_file: /home/isidora/IRCCS_Candiolo/Fastq/hg38.refGene.gtf #path to gene annotation file in GTF format
    exon_extension: no #extend exons by a certain width?
    extension_length: 0 #number of bp to extend exons by
    scaffold_length_min: 0 #minimal scaffold/chromosome length to consider (0 = all)
    additional_files: /home/isidora/IRCCS_Candiolo/Fastq/hg38.fa #Optional parameter. It is possible to give additonal reference sequences here, eg ERCC.fa
    additional_STAR_params: pGe.sjdbOverhang &gt; 0 #Optional parameter. you may add custom mapping parameters to STAR here
    #output directory
  out_dir: /home/isidora/IRCCS_Candiolo/Fastq/zUMIs/ #specify the full path to the output directory
  
  ###########################################
  #below, you may optionally change default parameters
  ###########################################
  
  #number of processors to use
  num_threads: 20
  mem_limit: null #Memory limit in Gigabytes, null meaning unlimited RAM usage
  #barcode &amp; UMI filtering options
  #number of bases under the base quality cutoff that should be filtered out.
  #Phred score base-cutoff for quality control.
  filter_cutoffs:
    BC_filter:
      num_bases: 3
      phred: 20
    UMI_filter:
      num_bases: 2
      phred: 20
  
  #Options for Barcode handling
  #You can give either number of top barcodes to use or give an annotation of cell barcodes
  #If you leave both barcode_num and barcode_file empty, zUMIs will perform automatic cell barcode selection for you!
  barcodes:
    barcode_num: null
    barcode_file: /home/isidora/IRCCS_Candiolo/Fastq/SampleSheet.txt
    barcode_sharing: null #Optional for combining several barcode sequences per cell (see github wiki)  
    automatic: yes #Give yes/no to this option. If the cell barcodes should be detected automatically. If the barcode file is given in combination with automatic barcode detection, the list of giiven barcodes will be used as whitelist.
    BarcodeBinning: 1 #Hamming distance binning of close cell barcode sequences.  
    nReadsperCell: 100 #Keep only the cell barcodes with atleast n number of reads.  
    demultiplex: yes #produce per-cell demultiplexed bam files.
  
  #Options related to counting of reads towards expression profiles
  counting_opts:
    introns: yes #can be set to no for exon-only counting.
    intronProb: no #perform an estimation of how likely intronic reads are to be derived from mRNA by comparing to intergenic counts.
    downsampling: '0' #Number of reads to downsample to. This value can be a fixed number of reads (e.g. 10000) or a desired range (e.g  10000-20000) Barcodes with less than &lt;d&gt; will not be reported. 0 means adaptive downsampling. Default: 0
    strand: 0 #Is the library stranded? 0 = unstranded, 1 = positively stranded, 2 = negatively stranded 
    Ham_Dist: 1 #Hamming distance collapsing of UMI sequences.
    velocyto: no #Would you like velocyto to do counting of intron-exon spanning reads  
    primaryHit: yes #Do you want to count the primary Hits of multimapping reads towards gene expression levels?
    multi_overlap: no #Do you want to assign reads overlapping to multiple features?  
    fraction_overlap: 0 #minimum required fraction of the read overlapping with the gene for read assignment to genes  
    twoPass: yes #perform basic STAR twoPass mapping
  
  #produce stats files and plots?
  make_stats: yes
  
  
  #Start zUMIs from stage. Possible TEXT(Filtering, Mapping, Counting,Summarising). Default: Filtering.
  which_Stage: Filtering
  
  #define dependencies program paths
  samtools_exec: samtools #samtools executable
  Rscript_exec: Rscript #Rscript executable
  STAR_exec: STAR #STAR executable
  pigz_exec: pigz #pigz executable
  
  #below, fqfilter will add a read_layout flag defining SE or PE
  </code></pre>
  <p>When I start the program with this code <code>zUMIs/zUMIs.sh -c -y ~/IRCCS_Candiolo/Fastq/zUMIs/zUMIs.yaml</code>, I get errors:</p>
  <pre><code>Using miniconda environment for zUMIs!
   note: internal executables will be used instead of those specified in the YAML file!
  
  
   You provided these parameters:
   YAML file:     /home/isidora/IRCCS_Candiolo/Fastq/zUMIs/zUMIs.yaml
   zUMIs directory:               /home/isidora/IRCCS_Candiolo/Fastq/zUMIs
   STAR executable                STAR
   samtools executable            samtools
   pigz executable                pigz
   Rscript executable             Rscript
   RAM limit:   null
   zUMIs version 2.9.7e
  
  
  Wed Aug 23 12:06:25 CEST 2023
  WARNING: The STAR version used for mapping is 2.7.3a and the STAR index was created using the version 2.7.4a. This may lead to an error while mapping. If you encounter any errors at the mapping stage, please make sure to create the STAR index using STAR 2.7.3a.
  Filtering...
  sh: 1: Syntax error: Unterminated quoted string
  sh: 1: Syntax error: Unterminated quoted string
  Wed Aug 23 12:06:26 CEST 2023
  Error in eval(bysub, parent.frame(), parent.frame()) :
    object 'XC' not found
  Calls: cellBC -&gt; [ -&gt; [.data.table -&gt; eval -&gt; eval
  In addition: Warning message:
  In data.table::fread(bccount_file, header = FALSE, col.names = c(&quot;XC&quot;,  :
    File '/home/isidora/IRCCS_Candiolo/Fastq/zUMIs//SmartSeq3.BCstats.txt' has size 0. Returning a NULL data.table.
  Execution halted
  Mapping...
  [1] &quot;2023-08-23 12:06:28 CEST&quot;
  Warning message:
  In data.table::fread(cmd = paste(samtools, &quot;view&quot;, filtered_bams[1],  :
    File '/tmp/RtmpKSSqU2/file57f5393a8fde' has size 0. Returning a NULL data.table.
  
  EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang &lt;=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
  SOLUTION: specify pGe.sjdbOverhang&gt;0, ideally readmateLength-1
  Aug 23 12:07:18 ...... FATAL ERROR, exiting
  Wed Aug 23 12:07:18 CEST 2023
  Counting...
  [1] &quot;2023-08-23 12:07:30 CEST&quot;
  Error in fread(paste0(opt$out_dir, &quot;/zUMIs_output/&quot;, opt$project, &quot;kept_barcodes_binned.txt&quot;)) :
    File '/home/isidora/IRCCS_Candiolo/Fastq/zUMIs//zUMIs_output/SmartSeq3kept_barcodes_binned.txt' does not exist or is non-readable. getwd()=='/home/isidora/IRCCS_Candiolo/Fastq/zUMIs'
  Execution halted
  Wed Aug 23 12:07:30 CEST 2023
  Loading required package: yaml
  Loading required package: Matrix
  [1] &quot;loomR found&quot;
  Error in gzfile(file, &quot;rb&quot;) : cannot open the connection
  Calls: rds_to_loom -&gt; readRDS -&gt; gzfile
  In addition: Warning message:
  In gzfile(file, &quot;rb&quot;) :
    cannot open compressed file '/home/isidora/IRCCS_Candiolo/Fastq/zUMIs//zUMIs_output/expression/SmartSeq3.dgecounts.rds', probable reason 'No such file or directory'
  Execution halted
  Wed Aug 23 12:07:32 CEST 2023
  Descriptive statistics...
  [1] &quot;I am loading useful packages for plotting...&quot;
  [1] &quot;2023-08-23 12:07:33 CEST&quot;
  Error in data.table::fread(paste0(opt$out_dir, &quot;/zUMIs_output/&quot;, opt$project,  :
    File '/home/isidora/IRCCS_Candiolo/Fastq/zUMIs//zUMIs_output/SmartSeq3kept_barcodes.txt' does not exist or is non-readable. getwd()=='/home/isidora/IRCCS_Candiolo/Fastq/zUMIs'
  Execution halted
  </code></pre>
  <p>I am new at bioinformatic field and I do not understand the meaning of these errors and the things I should correct.
  Can anyone help me please?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-23T14:50:31.183Z
Comments.Text: I don't see any obvious missing `'` or `"` (that's what "Unterminated quoted string" means) in the [zUmis.sh](https://github.com/sdparekh/zUMIs/blob/main/zUMIs.sh) version on github, but maybe yours is different? Can you changing `downsampling: '0'` to `downsampling: 0` in your yaml file and relaunching? I doubt this is the issue, but it's the only quote I see there and maybe the script isn't picking it up correctly (you normally don't need to quote integers, and this is the only quoted one).
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-23T14:54:53.273Z
Comments.Text: Also, should `STAR_index:` be pointing to a file instead of a directory? And did you see the warning? `WARNING: The STAR version used for mapping is 2.7.3a and the STAR index was created using the version 2.7.4a. This may lead to an error while mapping. If you encounter any errors at the mapping stage, please make sure to create the STAR index using STAR 2.7.3a.` Have you tried that?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-12T09:56:38.350Z
Comments.Text: Hay, it might be a bit late, but if you're still working with this I have some suggestions. 1. If you have multiple fastq.gz files, first follow this - https://github.com/sdparekh/zUMIs/wiki/Starting-from-demultiplexed-fastq-files then retry the pipeline.
Comments.UserDisplayName: Karthik Nair
Tags: <command-line><bash>
Title: Analysis of RNA sequencing with zUMIs
ThreadText: # Analysis of RNA sequencing with zUMIs
  
  (Asked by: username_0 on 2023-08-23)
  
  I wrote the .yaml file which was according the instruction from zUMIs:
  
  ```
  ###########################################
  #Welcome to zUMIs
  #below, please fill the mandatory inputs
  #We expect full paths for all files.
  ###########################################
  
  #define a project name that will be used to name output files
  project: SmartSeq3
  
  #Sequencing File Inputs:
  #For each input file, make one list object & define path and barcode ranges 
  #base definition vocabulary: BC(n) UMI(n) cDNA(n).
  #Barcode range definition needs to account for all ranges.You can  give several comma-separated ranges for BC & UMI sequences, eg.BC(1-6,20-26)
  #you can specify between 1 and 4 input files
  
  sequence_files:
    file1:
      name: /home/username_0/IRCCS_Candiolo/Fastq/*_R1_001_val_1.fq ##/home/username_0/IRCCS_Candiolo/Fastq/zUMIsFiles1/ #path to first file
      base_definition:
        - cDNA(1-150)
        - BC(1-8)
        - UMI(12-19)
    file2:
      name: /home/username_0/IRCCS_Candiolo/Fastq/*_R2_001_val_2.fq ## /home/username_0/IRCCS_Candiolo/Fastq/zUMIsFiles2/ #path to second file
      base_definition:
        - cDNA(1-150)
        - BC(1-8)
        - UMI(12-19)
  
  #reference genome setup
  reference:
    STAR_index: /home/username_0/IRCCS_Candiolo/Fastq/ #path to STAR genome index
    GTF_file: /home/username_0/IRCCS_Candiolo/Fastq/hg38.refGene.gtf #path to gene annotation file in GTF format
    exon_extension: no #extend exons by a certain width?
    extension_length: 0 #number of bp to extend exons by
    scaffold_length_min: 0 #minimal scaffold/chromosome length to consider (0 = all)
    additional_files: /home/username_0/IRCCS_Candiolo/Fastq/hg38.fa #Optional parameter. It is possible to give additonal reference sequences here, eg ERCC.fa
    additional_STAR_params: pGe.sjdbOverhang > 0 #Optional parameter. you may add custom mapping parameters to STAR here
    #output directory
  out_dir: /home/username_0/IRCCS_Candiolo/Fastq/zUMIs/ #specify the full path to the output directory
  
  ###########################################
  #below, you may optionally change default parameters
  ###########################################
  
  #number of processors to use
  num_threads: 20
  mem_limit: null #Memory limit in Gigabytes, null meaning unlimited RAM usage
  #barcode & UMI filtering options
  #number of bases under the base quality cutoff that should be filtered out.
  #Phred score base-cutoff for quality control.
  filter_cutoffs:
    BC_filter:
      num_bases: 3
      phred: 20
    UMI_filter:
      num_bases: 2
      phred: 20
  
  #Options for Barcode handling
  #You can give either number of top barcodes to use or give an annotation of cell barcodes
  #If you leave both barcode_num and barcode_file empty, zUMIs will perform automatic cell barcode selection for you!
  barcodes:
    barcode_num: null
    barcode_file: /home/username_0/IRCCS_Candiolo/Fastq/SampleSheet.txt
    barcode_sharing: null #Optional for combining several barcode sequences per cell (see github wiki)  
    automatic: yes #Give yes/no to this option. If the cell barcodes should be detected automatically. If the barcode file is given in combination with automatic barcode detection, the list of giiven barcodes will be used as whitelist.
    BarcodeBinning: 1 #Hamming distance binning of close cell barcode sequences.  
    nReadsperCell: 100 #Keep only the cell barcodes with atleast n number of reads.  
    demultiplex: yes #produce per-cell demultiplexed bam files.
  
  #Options related to counting of reads towards expression profiles
  counting_opts:
    introns: yes #can be set to no for exon-only counting.
    intronProb: no #perform an estimation of how likely intronic reads are to be derived from mRNA by comparing to intergenic counts.
    downsampling: '0' #Number of reads to downsample to. This value can be a fixed number of reads (e.g. 10000) or a desired range (e.g  10000-20000) Barcodes with less than <d> will not be reported. 0 means adaptive downsampling. Default: 0
    strand: 0 #Is the library stranded? 0 = unstranded, 1 = positively stranded, 2 = negatively stranded 
    Ham_Dist: 1 #Hamming distance collapsing of UMI sequences.
    velocyto: no #Would you like velocyto to do counting of intron-exon spanning reads  
    primaryHit: yes #Do you want to count the primary Hits of multimapping reads towards gene expression levels?
    multi_overlap: no #Do you want to assign reads overlapping to multiple features?  
    fraction_overlap: 0 #minimum required fraction of the read overlapping with the gene for read assignment to genes  
    twoPass: yes #perform basic STAR twoPass mapping
  
  #produce stats files and plots?
  make_stats: yes
  
  #Start zUMIs from stage. Possible TEXT(Filtering, Mapping, Counting,Summarising). Default: Filtering.
  which_Stage: Filtering
  
  #define dependencies program paths
  samtools_exec: samtools #samtools executable
  Rscript_exec: Rscript #Rscript executable
  STAR_exec: STAR #STAR executable
  pigz_exec: pigz #pigz executable
  
  #below, fqfilter will add a read_layout flag defining SE or PE
  
  ```
  
  When I start the program with this code `zUMIs/zUMIs.sh -c -y ~/IRCCS_Candiolo/Fastq/zUMIs/zUMIs.yaml`, I get errors:
  
  ```
  Using miniconda environment for zUMIs!
   note: internal executables will be used instead of those specified in the YAML file!
  
   You provided these parameters:
   YAML file:     /home/username_0/IRCCS_Candiolo/Fastq/zUMIs/zUMIs.yaml
   zUMIs directory:               /home/username_0/IRCCS_Candiolo/Fastq/zUMIs
   STAR executable                STAR
   samtools executable            samtools
   pigz executable                pigz
   Rscript executable             Rscript
   RAM limit:   null
   zUMIs version 2.9.7e
  
  Wed Aug 23 12:06:25 CEST 2023
  WARNING: The STAR version used for mapping is 2.7.3a and the STAR index was created using the version 2.7.4a. This may lead to an error while mapping. If you encounter any errors at the mapping stage, please make sure to create the STAR index using STAR 2.7.3a.
  Filtering...
  sh: 1: Syntax error: Unterminated quoted string
  sh: 1: Syntax error: Unterminated quoted string
  Wed Aug 23 12:06:26 CEST 2023
  Error in eval(bysub, parent.frame(), parent.frame()) :
    object 'XC' not found
  Calls: cellBC -> [ -> [.data.table -> eval -> eval
  In addition: Warning message:
  In data.table::fread(bccount_file, header = FALSE, col.names = c("XC",  :
    File '/home/username_0/IRCCS_Candiolo/Fastq/zUMIs//SmartSeq3.BCstats.txt' has size 0. Returning a NULL data.table.
  Execution halted
  Mapping...
  [1] "2023-08-23 12:06:28 CEST"
  Warning message:
  In data.table::fread(cmd = paste(samtools, "view", filtered_bams[1],  :
    File '/tmp/RtmpKSSqU2/file57f5393a8fde' has size 0. Returning a NULL data.table.
  
  EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
  SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
  Aug 23 12:07:18 ...... FATAL ERROR, exiting
  Wed Aug 23 12:07:18 CEST 2023
  Counting...
  [1] "2023-08-23 12:07:30 CEST"
  Error in fread(paste0(opt$out_dir, "/zUMIs_output/", opt$project, "kept_barcodes_binned.txt")) :
    File '/home/username_0/IRCCS_Candiolo/Fastq/zUMIs//zUMIs_output/SmartSeq3kept_barcodes_binned.txt' does not exist or is non-readable. getwd()=='/home/username_0/IRCCS_Candiolo/Fastq/zUMIs'
  Execution halted
  Wed Aug 23 12:07:30 CEST 2023
  Loading required package: yaml
  Loading required package: Matrix
  [1] "loomR found"
  Error in gzfile(file, "rb") : cannot open the connection
  Calls: rds_to_loom -> readRDS -> gzfile
  In addition: Warning message:
  In gzfile(file, "rb") :
    cannot open compressed file '/home/username_0/IRCCS_Candiolo/Fastq/zUMIs//zUMIs_output/expression/SmartSeq3.dgecounts.rds', probable reason 'No such file or directory'
  Execution halted
  Wed Aug 23 12:07:32 CEST 2023
  Descriptive statistics...
  [1] "I am loading useful packages for plotting..."
  [1] "2023-08-23 12:07:33 CEST"
  Error in data.table::fread(paste0(opt$out_dir, "/zUMIs_output/", opt$project,  :
    File '/home/username_0/IRCCS_Candiolo/Fastq/zUMIs//zUMIs_output/SmartSeq3kept_barcodes.txt' does not exist or is non-readable. getwd()=='/home/username_0/IRCCS_Candiolo/Fastq/zUMIs'
  Execution halted
  
  ```
  
  I am new at bioinformatic field and I do not understand the meaning of these errors and the things I should correct.
  Can anyone help me please?
  
  **Tags:** command-line, bash
  
  > **Comments:**
  > **username_1**: I don't see any obvious missing `'` or `"` (that's what "Unterminated quoted string" means) in the [zUmis.sh](https://github.com/sdparekh/zUMIs/blob/main/zUMIs.sh) version on github, but maybe yours is different? Can you changing `downsampling: '0'` to `downsampling: 0` in your yaml file and relaunching? I doubt this is the issue, but it's the only quote I see there and maybe the script isn't picking it up correctly (you normally don't need to quote integers, and this is the only quoted one).  
  > **username_1**: Also, should `STAR_index:` be pointing to a file instead of a directory? And did you see the warning? `WARNING: The STAR version used for mapping is 2.7.3a and the STAR index was created using the version 2.7.4a. This may lead to an error while mapping. If you encounter any errors at the mapping stage, please make sure to create the STAR index using STAR 2.7.3a.` Have you tried that?  
  > **username_2**: Hay, it might be a bit late, but if you're still working with this I have some suggestions. 1. If you have multiple fastq.gz files, first follow this - https://github.com/sdparekh/zUMIs/wiki/Starting-from-demultiplexed-fastq-files then retry the pipeline.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-08-24. Score: 3
  
  I'd advise you to keep away from ZUMIs unless you're doing the same workflow that it was designed for. I spent many days trying to sort out issues with that program, and eventually gave up; it wasn't created with the expectation of errors or unexpected input, and had too many programming minefields for the data I was working with.
  
  ZUMIs should be stopping on the first error and explaining what's wrong. As demonstrated by your situation, it's not; it's trying to keep on processing, barreling through and dragging errors along for the ride. It's not exactly beginner-friendly code.
  
  In the end, I changed to using Salmon/Alevin, which had a more fail-resistant interface, and (eventually) similar barcode flexibility:
  
  <https://combine-lab.github.io/alevin-fry-tutorials/2021/custom-geo/>
  
  > **Comments:**
  > **username_0**: 0   Thank you for your quick answer. I did the SmartSeq 3 protocol and it is advised to do bioinformatic analysis with zUMIs pipeline because of the specificities of results. Do you know is the Salmon/Alevin suitable for this protocol?  
  > **username_3**: I don't know, sorry. If you've been advised to use ZUMIs, then do that, and try to get more help from the person who gave you that advice.  
  
  ---
  
  