Answers.Body: <p>The issue was caused by the corrupted input <code>.fastq</code> files that were damaged somehow upon uploading them from the local machine. We figured it out by <code>md5sum</code> command output comparison. We have not yet managed to actually run all of the script still, it is failing, but for another reason now. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T22:39:22.913Z
Answers.LastActivityDate: 2017-12-14T22:39:22.913Z
Answers.UserDisplayName: Nikita Vlasenko
Body: <p>I am trying to run salmon and it keeps giving me 2 java exceptions:</p>
  
  <pre><code>java.io.EOFException
      at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268)
      at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
      at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
      at java.util.zip.GZIPInputStream.&lt;init&gt;(GZIPInputStream.java:79)
      at java.util.zip.GZIPInputStream.&lt;init&gt;(GZIPInputStream.java:91)
      at uk.ac.babraham.FastQC.Utilities.MultiMemberGZIPInputStream.&lt;init&gt;(MultiMemberGZIPInputStream.java:37)
  </code></pre>
  
  <p>and </p>
  
  <pre><code>java.lang.ArrayIndexOutOfBoundsException: -1
      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:100)
      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:184)
      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.&lt;init&gt;(HTMLReportArchive.java:84)
      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
      at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
      at java.lang.Thread.run(Thread.java:745)
  
      at uk.ac.babraham.FastQC.Sequence.FastQFile.&lt;init&gt;(FastQFile.java:80)
      at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:106)
      at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:62)
      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.processFile(OfflineRunner.java:129)
      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.&lt;init&gt;(OfflineRunner.java:102)
      at uk.ac.babraham.FastQC.FastQCApplication.main(FastQCApplication.java:316)
  </code></pre>
  
  <p>I installed <code>salmon</code> using <code>Anaconda</code>s' <code>conda install -c bioconda salmon</code> and all other necessary packages in the same way. I thought that it could be a problem with <code>fastqc</code>, so I uninstalled it and then installed it manually (through the fastqc<code>.zip</code> file), but the output remained the same.
  I am using <code>Java8</code>:</p>
  
  <pre><code>java -version
  openjdk version "1.8.0_92"
  OpenJDK Runtime Environment (Zulu 8.15.0.1-linux64) (build 1.8.0_92-b15)
  OpenJDK 64-Bit Server VM (Zulu 8.15.0.1-linux64) (build 25.92-b15, mixed mode)
  </code></pre>
  
  <p>I am running the script on a cluster with <code>SLURM</code>. Locally on my laptop the script is running without any issues. I am really stuck and do not know what to try, so any suggestions would be greatly appreciated.</p>
  
  <p><code>Update</code>
  Another issue that I just found is the following (the last command that <code>salmon</code> is trying to execute):</p>
  
  <pre><code>Exception : [Failed to read 8 bytes from input stream! Read 0]
  salmon quant was invoked improperly.
  </code></pre>
  
  <p>Then I searched online for the issue and found the following thread:</p>
  
  <p><a href="https://github.com/COMBINE-lab/salmon/issues/129" rel="nofollow noreferrer">https://github.com/COMBINE-lab/salmon/issues/129</a></p>
  
  <p>However, updating <code>salmon</code> to the newest <code>0.9.1</code> version did not solve the issue.</p>
  
  <p><code>Edit</code>
  The command I am running is the following one:</p>
  
  <pre><code>python "$path/salmon_rna_seq.py" -L 'IU' -E "$path/jyvo_experiment-
  metadata.yaml" -T 'gencode_mouse_m13' -C "$path/salmon_rna_seq.yaml" -I 
  "$path_to_fastq/$name-lec_1_r1.fq.gz" "$path_to_fastq/$name-
  lec_1_r2.fq.gz" -O "$path/res/$name-lec_1" -S "$name-lec_1" -P '6' -X 'True'
  </code></pre>
  
  <p>The command is run by <code>Pypiper</code>: <a href="https://github.com/epigen/pypiper" rel="nofollow noreferrer">https://github.com/epigen/pypiper</a>
  Inside of <code>salmon_rna_seq.py</code> I am also running commands to treat the data, the <code>salmon</code> part looks the following way:</p>
  
  <pre><code>def salmon(self):
      cmd = "salmon quant -i " + self.indexed_transcriptome
      cmd += " -l " + self.lib_type
  
      if self.trim:
          cmd += " -1 " + self.sample_files['r1']['fastq']['trimmed_paired']
          cmd += " -2 " + self.sample_files['r2']['fastq']['trimmed_paired']
      else:
          cmd += " -1 " + self.sample_files['r1']['fastq']['chastity']
          cmd += " -2 " + self.sample_files['r2']['fastq']['chastity']
  
      cmd += " -o " + self.sample_dirs['salmon']['base']
      cmd += " --numBootstraps=30"
  self.pipe_manager.run(cmd, self.sample_files['salmon'])
  </code></pre>
  
  <p>Commands that the <code>Pypiper</code> runs are the following ones (<code>_commands.sh</code> <code>Pypiper</code> file): </p>
  
  <pre><code>zcat -c $PATH/res/old-lec_1/fastq/merged/old-lec_1_r1.fq.gz | grep -A 3 
  '^@.*[^:]*:N:[^:]*:' | grep -v '^\-\-$' | gzip -c &gt; $PATH/res/old-lec_1/fastq
  /chastity/old-lec_1_r1.fq.gz
  
  zcat -c $PATH/res/old-lec_1/fastq/merged/old-lec_1_r2.fq.gz | grep -A 3 
  '^@.*[^:]*:N:[^:]*:' | grep -v '^\-\-$' | gzip -c &gt; $PATH/res/old-lec_1/fastq
  /chastity/old-lec_1_r2.fq.gz
  
  java -jar $HOME/anaconda2/share/trimmomatic-0.36-5
  /trimmomatic.jar PE -threads 6 -phred33 $PATH/res/old-lec_1/fastq/chastity/old-
  lec_1_r1.fq.gz $PATH/res/old-lec_1/fastq/chastity/old-lec_1_r2.fq.gz $PATH/res/old-
  lec_1/fastq/trimmed/old-lec_1_r1.fq.gz $PATH/res/old-lec_1/fastq/trimmed/unpaired
  /old-lec_1_r1.fq.gz $PATH/res/old-lec_1/fastq/trimmed/old-lec_1_r2.fq.gz $PATH/res/old-
  lec_1/fastq/trimmed/unpaired/old-lec_1_r2.fq.gz ILLUMINACLIP:$HOME/anaconda2/share/trimmomatic-0.36-5/adapters
  /Small_List_Of_Adapters.fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15
  MINLEN:36
  
  fastqc $PATH/res/old-lec_1/fastq/merged/old-lec_1_r1.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/merged
  
  fastqc $PATH/res/old-lec_1/fastq/chastity/old-lec_1_r1.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/chastity
  
  fastqc $PATH/res/old-lec_1/fastq/merged/old-lec_1_r2.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/merged
  
  fastqc $PATH/res/old-lec_1/fastq/chastity/old-lec_1_r2.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/chastity
  
  fastqc $PATH/res/old-lec_1/fastq/trimmed/old-lec_1_r1.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/trimmed
  
  fastqc $PATH/res/old-lec_1/fastq/trimmed/old-lec_1_r2.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/trimmed
  
  salmon quant -i $HOME/Genomes/mouse/salmon
  /gencode.vM13.transcripts.fa.sidx -l IU -1 $PATH/res/old-lec_1/fastq/trimmed
  /old-lec_1_r1.fq.gz -2 $PATH/res/old-lec_1/fastq/trimmed/old-
  lec_1_r2.fq.gz -o $PATH/res/old-lec_1/quantification --numBootstraps=30
  </code></pre>
  
  <p><code>Edit</code></p>
  
  <p>After running manually <code>gzip -t</code> on a file:</p>
  
  <pre><code>-bash-4.2$ gzip -t old-lec_1_r1.fq.gz 
  gzip: old-lec_1_r1.fq.gz: unexpected end of file
  </code></pre>
  
  <p>So, I guess the file is corrupted and the issue is with some library that is generating the file. However, reinstalling manually other libraries do not help.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-12T23:57:55.197Z
Comments.Text: Can you run `gzip -t` on the file(s) to see if it/they are corrupt? That'd be the interpretation of the error messages you're seeing. Make sure to run this on the cluster, in case there's some sort of "modified in transit" networking error going on.
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T03:53:10.907Z
Comments.Text: This would be my guess.  Also, however, I'm not sure how Java is involved (since Salmon is written completely in C++).
Comments.UserDisplayName: nomad
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T05:06:14.143Z
Comments.Text: It looks like FastQC is being invoked somewhere. Providing the precise commands invoked will help us troubleshoot the problem.
Comments.UserDisplayName: Daniel Standage
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T09:59:00.550Z
Comments.Text: Yes, please [edit] your question and show the exact commands you are using. It's hard to debug something if we don't know how to replicate it.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T10:46:12.103Z
Comments.Text: Salmon is a red herring. The error message is clearly unrelated to Salmon.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T11:11:39.013Z
Comments.Text: Oh god. I swear I didn’t notice the pun in my previous comment.
Comments.UserDisplayName: Konrad Rudolph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T14:06:45.277Z
Comments.Text: +1 Konrad, for the unintentional pun ;P.
Comments.UserDisplayName: nomad
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T18:24:32.033Z
Comments.Text: Where does old-lec_1_r1.fq.gz come from?  That is the issue.
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-13T19:04:58.490Z
Comments.Text: FastQC is involved in generating it mainly
Comments.UserDisplayName: Nikita Vlasenko
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-14T08:05:49.253Z
Comments.Text: FastQC has absolutely nothing to do with generating the file, it just produces some QC metrics. It's unclear (A) why you're running grep on the files and (B) which of the multiple copies of old-lec_1_r1.fq.gz is the first that's corrupt.
Comments.UserDisplayName: Devon Ryan
Tags: <rna-seq><quantification><salmon>
Title: Exception : [Failed to read 8 bytes from input stream! Read 0] salmon quant was invoked improperly
ThreadText: # Exception : [Failed to read 8 bytes from input stream! Read 0] salmon quant was invoked improperly
  
  (Asked by: username_0 on 2017-12-12)
  
  I am trying to run salmon and it keeps giving me 2 java exceptions:
  
  ```
  java.io.EOFException
      at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268)
      at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)
      at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
      at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
      at uk.ac.babraham.FastQC.Utilities.MultiMemberGZIPInputStream.<init>(MultiMemberGZIPInputStream.java:37)
  
  ```
  
  and
  
  ```
  java.lang.ArrayIndexOutOfBoundsException: -1
      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:100)
      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:184)
      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
      at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
      at java.lang.Thread.run(Thread.java:745)
  
      at uk.ac.babraham.FastQC.Sequence.FastQFile.<init>(FastQFile.java:80)
      at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:106)
      at uk.ac.babraham.FastQC.Sequence.SequenceFactory.getSequenceFile(SequenceFactory.java:62)
      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.processFile(OfflineRunner.java:129)
      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.<init>(OfflineRunner.java:102)
      at uk.ac.babraham.FastQC.FastQCApplication.main(FastQCApplication.java:316)
  
  ```
  
  I installed `salmon` using `Anaconda`s' `conda install -c bioconda salmon` and all other necessary packages in the same way. I thought that it could be a problem with `fastqc`, so I uninstalled it and then installed it manually (through the fastqc`.zip` file), but the output remained the same.
  I am using `Java8`:
  
  ```
  java -version
  openjdk version "1.8.0_92"
  OpenJDK Runtime Environment (Zulu 8.15.0.1-linux64) (build 1.8.0_92-b15)
  OpenJDK 64-Bit Server VM (Zulu 8.15.0.1-linux64) (build 25.92-b15, mixed mode)
  
  ```
  
  I am running the script on a cluster with `SLURM`. Locally on my laptop the script is running without any issues. I am really stuck and do not know what to try, so any suggestions would be greatly appreciated.
  
  `Update`
  Another issue that I just found is the following (the last command that `salmon` is trying to execute):
  
  ```
  Exception : [Failed to read 8 bytes from input stream! Read 0]
  salmon quant was invoked improperly.
  
  ```
  
  Then I searched online for the issue and found the following thread:
  
  <https://github.com/COMBINE-lab/salmon/issues/129>
  
  However, updating `salmon` to the newest `0.9.1` version did not solve the issue.
  
  `Edit`
  The command I am running is the following one:
  
  ```
  python "$path/salmon_rna_seq.py" -L 'IU' -E "$path/jyvo_experiment-
  metadata.yaml" -T 'gencode_mouse_m13' -C "$path/salmon_rna_seq.yaml" -I 
  "$path_to_fastq/$name-lec_1_r1.fq.gz" "$path_to_fastq/$name-
  lec_1_r2.fq.gz" -O "$path/res/$name-lec_1" -S "$name-lec_1" -P '6' -X 'True'
  
  ```
  
  The command is run by `Pypiper`: <https://github.com/epigen/pypiper>
  Inside of `salmon_rna_seq.py` I am also running commands to treat the data, the `salmon` part looks the following way:
  
  ```
  def salmon(self):
      cmd = "salmon quant -i " + self.indexed_transcriptome
      cmd += " -l " + self.lib_type
  
      if self.trim:
          cmd += " -1 " + self.sample_files['r1']['fastq']['trimmed_paired']
          cmd += " -2 " + self.sample_files['r2']['fastq']['trimmed_paired']
      else:
          cmd += " -1 " + self.sample_files['r1']['fastq']['chastity']
          cmd += " -2 " + self.sample_files['r2']['fastq']['chastity']
  
      cmd += " -o " + self.sample_dirs['salmon']['base']
      cmd += " --numBootstraps=30"
  self.pipe_manager.run(cmd, self.sample_files['salmon'])
  
  ```
  
  Commands that the `Pypiper` runs are the following ones (`_commands.sh` `Pypiper` file):
  
  ```
  zcat -c $PATH/res/old-lec_1/fastq/merged/old-lec_1_r1.fq.gz | grep -A 3 
  '^@.*[^:]*:N:[^:]*:' | grep -v '^\-\-$' | gzip -c > $PATH/res/old-lec_1/fastq
  /chastity/old-lec_1_r1.fq.gz
  
  zcat -c $PATH/res/old-lec_1/fastq/merged/old-lec_1_r2.fq.gz | grep -A 3 
  '^@.*[^:]*:N:[^:]*:' | grep -v '^\-\-$' | gzip -c > $PATH/res/old-lec_1/fastq
  /chastity/old-lec_1_r2.fq.gz
  
  java -jar $HOME/anaconda2/share/trimmomatic-0.36-5
  /trimmomatic.jar PE -threads 6 -phred33 $PATH/res/old-lec_1/fastq/chastity/old-
  lec_1_r1.fq.gz $PATH/res/old-lec_1/fastq/chastity/old-lec_1_r2.fq.gz $PATH/res/old-
  lec_1/fastq/trimmed/old-lec_1_r1.fq.gz $PATH/res/old-lec_1/fastq/trimmed/unpaired
  /old-lec_1_r1.fq.gz $PATH/res/old-lec_1/fastq/trimmed/old-lec_1_r2.fq.gz $PATH/res/old-
  lec_1/fastq/trimmed/unpaired/old-lec_1_r2.fq.gz ILLUMINACLIP:$HOME/anaconda2/share/trimmomatic-0.36-5/adapters
  /Small_List_Of_Adapters.fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15
  MINLEN:36
  
  fastqc $PATH/res/old-lec_1/fastq/merged/old-lec_1_r1.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/merged
  
  fastqc $PATH/res/old-lec_1/fastq/chastity/old-lec_1_r1.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/chastity
  
  fastqc $PATH/res/old-lec_1/fastq/merged/old-lec_1_r2.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/merged
  
  fastqc $PATH/res/old-lec_1/fastq/chastity/old-lec_1_r2.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/chastity
  
  fastqc $PATH/res/old-lec_1/fastq/trimmed/old-lec_1_r1.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/trimmed
  
  fastqc $PATH/res/old-lec_1/fastq/trimmed/old-lec_1_r2.fq.gz --outdir $PATH/res/old-
  lec_1/fastqc/trimmed
  
  salmon quant -i $HOME/Genomes/mouse/salmon
  /gencode.vM13.transcripts.fa.sidx -l IU -1 $PATH/res/old-lec_1/fastq/trimmed
  /old-lec_1_r1.fq.gz -2 $PATH/res/old-lec_1/fastq/trimmed/old-
  lec_1_r2.fq.gz -o $PATH/res/old-lec_1/quantification --numBootstraps=30
  
  ```
  
  `Edit`
  
  After running manually `gzip -t` on a file:
  
  ```
  -bash-4.2$ gzip -t old-lec_1_r1.fq.gz 
  gzip: old-lec_1_r1.fq.gz: unexpected end of file
  
  ```
  
  So, I guess the file is corrupted and the issue is with some library that is generating the file. However, reinstalling manually other libraries do not help.
  
  **Tags:** rna-seq, quantification, salmon
  
  > **Comments:**
  > **username_1**: Can you run `gzip -t` on the file(s) to see if it/they are corrupt? That'd be the interpretation of the error messages you're seeing. Make sure to run this on the cluster, in case there's some sort of "modified in transit" networking error going on.  
  > **username_2**: This would be my guess.  Also, however, I'm not sure how Java is involved (since Salmon is written completely in C++).  
  > **username_3**: It looks like FastQC is being invoked somewhere. Providing the precise commands invoked will help us troubleshoot the problem.  
  > **username_4**: Yes, please [edit] your question and show the exact commands you are using. It's hard to debug something if we don't know how to replicate it.  
  > **username_5**: Salmon is a red herring. The error message is clearly unrelated to Salmon.  
  > **username_5**: Oh god. I swear I didn’t notice the pun in my previous comment.  
  > **username_2**: +1 username_5, for the unintentional pun ;P.  
  > **username_6**: Where does old-lec_1_r1.fq.gz come from?  That is the issue.  
  > **username_0**: FastQC is involved in generating it mainly  
  > **username_1**: FastQC has absolutely nothing to do with generating the file, it just produces some QC metrics. It's unclear (A) why you're running grep on the files and (B) which of the multiple copies of old-lec_1_r1.fq.gz is the first that's corrupt.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2017-12-14. Score: 1 | Accepted answer
  
  The issue was caused by the corrupted input `.fastq` files that were damaged somehow upon uploading them from the local machine. We figured it out by `md5sum` command output comparison. We have not yet managed to actually run all of the script still, it is failing, but for another reason now.
  
  
  ---
  
  