Answers.Body: <p>It's possible you're looking at instructions for a recent version of samtools / bcftools, whereas your computer has an older version. Can you try <code>samtools mpileup -g -f ...</code> to output the pileup as BCF format instead of <code>bcftools mpileup -f ...</code>?</p>
  
  <p>Update: it's good to know that the samtools / bcftools pipelines are at least producing consistent results, which suggests that the error is not associated with the output format on the mpileup side of the pipeline, and also good to see that you've identified that this was a memory issue.</p>
  
  <p>The most frequent cause of memory issues I've had with sorted BAM files has been trying to figure out variants in extremely high coverage data (e.g. over 1000X), which would make sense for a bacterial isolate sequenced on a GridION. I use <a href="https://gitlab.com/gringer/bioinfscripts/blob/master/samNormalise.pl" rel="nofollow noreferrer">my own script</a> to carry out a reservoir sampling on mapped BAM format files, which reduces coverage down to a more manageable level. Here's an example command to normalise to 100X coverage where possible:</p>
  
  <pre><code>samtools view reads.bam | ./samNormalise.pl -coverage 100 | \
     samtools view -b &gt; reads_normalised.bam
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-09T07:22:19.383Z
Answers.Text: did not work unfortunately, got the following messages after calling:
  `samtools mpileup -g -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz`
  
  `[warning] samtools mpileup option `g` is functional, but deprecated. Please switch to using bcftools mpileup in future.`
  
  `Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid`
  
  `[mpileup] 1 samples in 1 input files`
  
  `Failed to open -: unknown file type`
Answers.UserDisplayName: BCArg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-09T07:40:07.633Z
Answers.Text: The VCF/BCF-outputting `mpileup` command was transferred from samtools to bcftools relatively recently. So in fact it is @gringer's suggestion that is instructions for an earlier version of samtools / bcftools.
Answers.UserDisplayName: John Marshall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-09T05:43:51.213Z
Answers.LastActivityDate: 2019-07-09T14:35:02.110Z
Answers.LastEditDate: 2019-07-09T14:35:02.110Z
Answers.UserDisplayName: gringer
Answers.Body: <p>The <code>-</code> is usually used to mean standard input when reading data. So the message would suggest that the second <code>bcftools</code> command fails to read from stdin, so it fails to read the output of the first <code>bcftools</code> command:</p>
  
  <pre><code>$ echo "foo bar" | /ref/software/bcftools-1.8/bcftools call -mv -Oz -o calls.vcf.gz
  Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
  Failed to open -: unknown file type
  </code></pre>
  
  <p>So there is something wrong with either your input file or with the output produced when running the first <code>bcftools</code> command on that input file. We can't really help more unless you share the file. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-09T07:53:07.547Z
Answers.Text: I doubt there's any problem with the input files, as I can visualise the alignment on table or IGV, so I guess the `bcftoools mpileup` is not generating any output (please see my comment to @John Marshall). I guess I will report the problem on their github page.
Answers.UserDisplayName: BCArg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-09T07:56:48.707Z
Answers.Text: @BCArg yes, precisely. If you got `Killed` in the terminal, then the process is being killed by the operating system, most likely because it is taking up too much memory. Sounds like you need to run this on a more powerful machine.
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-09T08:12:59.330Z
Answers.Text: yes mystery solved, I monitored the the performance of the command `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp` with `htop` and the program got killed right after I reached the maximum of my RAM memory (16GB), so in fact running the command on the cluster appears to be the solution.
Answers.UserDisplayName: BCArg
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-08T16:15:40.243Z
Answers.LastActivityDate: 2019-07-08T16:15:40.243Z
Answers.UserDisplayName: terdon
Body: <p>I have sequenced a bacterial genome with a GridIon from ONT. Basically what I want to check is whether or not trimming 50 bps at the beginning of the reads will improve alignment against the reference genome and ultimately the call of a consensus sequence.</p>
  
  <p>For that what I have done so far is:</p>
  
  <ol>
  <li>concatenated all the fastqs generated from the Gridion into a single (very large) fastq file: <code>ref494_cat_all.fastq</code></li>
  <li>aligned this concatenated fastq against the reference genome <code>ref_genbank.fasta</code> with <code>minimap2</code>: <code>~/Downloads/minimap2-2.17_x64-linux/minimap2 -ax map-ont ref_genbank/ref_genbank.fasta ref494_cat_all.fastq &gt; ref494_aligned_trim50.sam</code></li>
  <li>Performed the general <code>samtools</code> steps to covert sam to bam, index and sort</li>
  </ol>
  
  <p>Then, in order to get the consensus sequence I first try to generate a <code>.vcf.gz</code> file, according to <a href="https://samtools.github.io/bcftools/howtos/variant-calling.html" rel="nofollow noreferrer">this tutorial</a></p>
  
  <pre><code>bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz
  </code></pre>
  
  <p>Immediatelly after calling the command above I got:</p>
  
  <pre><code>Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
  [mpileup] 1 samples in 1 input files
  </code></pre>
  
  <p>Then, after a minute or two, I got:</p>
  
  <pre><code>Failed to open -: unknown file type
  </code></pre>
  
  <p>I'm pretty sure that there's no issue with <code>.bam</code> file, as I could visualise the alignment with IGV or tablet.</p>
  
  <p>Does anyone know what the issue can be?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T07:28:17.813Z
Comments.Text: Please capture the output of the first command and show us the first few lines (via `bcftools view` if it's not text): `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp`
Comments.UserDisplayName: John Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T07:49:30.853Z
Comments.Text: I got `Killed` output to the terminal after a minute or two. Also this command generates an empty `output.tmp` file.
Comments.UserDisplayName: BCArg
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T09:12:31.937Z
Comments.Text: As noted below, killed due to memory exhaustion. However: you deserve an error message. Mpileup typically doesn't use that much memory and I have been unable to reproduce this on (non-ONT) data here. If you are able to, it would be interesting to try current **develop** bcftools to see if this still happens and/or to run it under the debugger and see where it crashes.
Comments.UserDisplayName: John Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T09:17:38.730Z
Comments.Text: The latter is quite easy: run `gdb --args bcftools mpileup -f ref_genbank/ref_genbank.fasta -o /dev/null ref494_aligned.sorted.bam`, then type `run` and wait for it to crash. Then type `backtrace`; the output from this will be instructive.
Comments.UserDisplayName: John Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T09:58:42.097Z
Comments.Text: Just before the program got killed I got `Program terminated with signal SIGKILL, Killed.` `The program no longer exists.`
  
  The output of `backtrace` was `No stack`
  
  That you cannot reproduce this error on non-ONT data, you think it can be related to the size of my input files? My bam is 14GB (coverage is about 2.000, we ran for about 48h, which we won't do routinely)
Comments.UserDisplayName: BCArg
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T10:07:31.197Z
Comments.Text: Oh yeah, SIGKILL. Perhaps you were hit by the OOM Killer (cf https://stackoverflow.com/questions/12288550/c-linux-binary-terminated-with-signal-sigkill-why-loaded-in-gdb) so the lack of error message maybe isn't bcftools's fault. 14G isn't a big BAM file, but the level of coverage and nature of ONT data (long reads with maybe interesting CIGARs) would be things that could make mpileup take more than the usual amount of memory.
Comments.UserDisplayName: John Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-09T13:42:36.107Z
Comments.Text: Just tried to run the original command (as in my question) on a cluster (24 cores, 48GB of RAM). It ran for an hour or so, then I got the same error `Failed to open -: unknown file type`, so I guess I will try some commands without piping to `bcftools` to see what output I will get
Comments.UserDisplayName: BCArg
Tags: <phylogenetics><sam><vcf><samtools><nanopore>
Title: Error while calling bcftools mpileup - Failed to open -: unknown file type
ThreadText: # Error while calling bcftools mpileup - Failed to open -: unknown file type
  
  (Asked by: username_0 on 2019-07-08)
  
  I have sequenced a bacterial genome with a GridIon from ONT. Basically what I want to check is whether or not trimming 50 bps at the beginning of the reads will improve alignment against the reference genome and ultimately the call of a consensus sequence.
  
  For that what I have done so far is:
  
  1. concatenated all the fastqs generated from the Gridion into a single (very large) fastq file: `ref494_cat_all.fastq`
  2. aligned this concatenated fastq against the reference genome `ref_genbank.fasta` with `minimap2`: `~/Downloads/minimap2-2.17_x64-linux/minimap2 -ax map-ont ref_genbank/ref_genbank.fasta ref494_cat_all.fastq > ref494_aligned_trim50.sam`
  3. Performed the general `samtools` steps to covert sam to bam, index and sort
  
  Then, in order to get the consensus sequence I first try to generate a `.vcf.gz` file, according to [this tutorial](https://samtools.github.io/bcftools/howtos/variant-calling.html)
  
  ```
  bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz
  
  ```
  
  Immediatelly after calling the command above I got:
  
  ```
  Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
  [mpileup] 1 samples in 1 input files
  
  ```
  
  Then, after a minute or two, I got:
  
  ```
  Failed to open -: unknown file type
  
  ```
  
  I'm pretty sure that there's no issue with `.bam` file, as I could visualise the alignment with IGV or tablet.
  
  Does anyone know what the issue can be?
  
  **Tags:** phylogenetics, sam, vcf, samtools, nanopore
  
  > **Comments:**
  > **username_1**: Please capture the output of the first command and show us the first few lines (via `bcftools view` if it's not text): `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp`  
  > **username_0**: I got `Killed` output to the terminal after a minute or two. Also this command generates an empty `output.tmp` file.  
  > **username_1**: As noted below, killed due to memory exhaustion. However: you deserve an error message. Mpileup typically doesn't use that much memory and I have been unable to reproduce this on (non-ONT) data here. If you are able to, it would be interesting to try current **develop** bcftools to see if this still happens and/or to run it under the debugger and see where it crashes.  
  > **username_1**: The latter is quite easy: run `gdb --args bcftools mpileup -f ref_genbank/ref_genbank.fasta -o /dev/null ref494_aligned.sorted.bam`, then type `run` and wait for it to crash. Then type `backtrace`; the output from this will be instructive.  
  > **username_0**: Just before the program got killed I got `Program terminated with signal SIGKILL, Killed.` `The program no longer exists.`  The output of `backtrace` was `No stack`  That you cannot reproduce this error on non-ONT data, you think it can be related to the size of my input files? My bam is 14GB (coverage is about 2.000, we ran for about 48h, which we won't do routinely)  
  > **username_1**: Oh yeah, SIGKILL. Perhaps you were hit by the OOM Killer (cf https://stackoverflow.com/questions/12288550/c-linux-binary-terminated-with-signal-sigkill-why-loaded-in-gdb) so the lack of error message maybe isn't bcftools's fault. 14G isn't a big BAM file, but the level of coverage and nature of ONT data (long reads with maybe interesting CIGARs) would be things that could make mpileup take more than the usual amount of memory.  
  > **username_0**: Just tried to run the original command (as in my question) on a cluster (24 cores, 48GB of RAM). It ran for an hour or so, then I got the same error `Failed to open -: unknown file type`, so I guess I will try some commands without piping to `bcftools` to see what output I will get  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-07-09. Score: 1
  
  It's possible you're looking at instructions for a recent version of samtools / bcftools, whereas your computer has an older version. Can you try `samtools mpileup -g -f ...` to output the pileup as BCF format instead of `bcftools mpileup -f ...`?
  
  Update: it's good to know that the samtools / bcftools pipelines are at least producing consistent results, which suggests that the error is not associated with the output format on the mpileup side of the pipeline, and also good to see that you've identified that this was a memory issue.
  
  The most frequent cause of memory issues I've had with sorted BAM files has been trying to figure out variants in extremely high coverage data (e.g. over 1000X), which would make sense for a bacterial isolate sequenced on a GridION. I use [my own script](https://gitlab.com/username_2/bioinfscripts/blob/master/samNormalise.pl) to carry out a reservoir sampling on mapped BAM format files, which reduces coverage down to a more manageable level. Here's an example command to normalise to 100X coverage where possible:
  
  ```
  samtools view reads.bam | ./samNormalise.pl -coverage 100 | \
     samtools view -b > reads_normalised.bam
  
  ```
  
  > **Comments:**
  > **username_0**: did not work unfortunately, got the following messages after calling: `samtools mpileup -g -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam | bcftools call -mv -Oz -o calls.vcf.gz`  `[warning] samtools mpileup option `g` is functional, but deprecated. Please switch to using bcftools mpileup in future.`  `Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid`  `[mpileup] 1 samples in 1 input files`  `Failed to open -: unknown file type`  
  > **username_1**: The VCF/BCF-outputting `mpileup` command was transferred from samtools to bcftools relatively recently. So in fact it is @username_2's suggestion that is instructions for an earlier version of samtools / bcftools.  
  
  ---
  
  ### Answer by username_3 on 2019-07-08. Score: 1
  
  The `-` is usually used to mean standard input when reading data. So the message would suggest that the second `bcftools` command fails to read from stdin, so it fails to read the output of the first `bcftools` command:
  
  ```
  $ echo "foo bar" | /ref/software/bcftools-1.8/bcftools call -mv -Oz -o calls.vcf.gz
  Note: none of --samples-file, --ploidy or --ploidy-file given, assuming all sites are diploid
  Failed to open -: unknown file type
  
  ```
  
  So there is something wrong with either your input file or with the output produced when running the first `bcftools` command on that input file. We can't really help more unless you share the file.
  
  > **Comments:**
  > **username_0**: I doubt there's any problem with the input files, as I can visualise the alignment on table or IGV, so I guess the `bcftoools mpileup` is not generating any output (please see my comment to @username_1). I guess I will report the problem on their github page.  
  > **username_3**: @username_0 yes, precisely. If you got `Killed` in the terminal, then the process is being killed by the operating system, most likely because it is taking up too much memory. Sounds like you need to run this on a more powerful machine.  
  > **username_0**: yes mystery solved, I monitored the the performance of the command `bcftools mpileup -f ref_genbank/ref_genbank.fasta ref494_aligned.sorted.bam > output.tmp` with `htop` and the program got killed right after I reached the maximum of my RAM memory (16GB), so in fact running the command on the cluster appears to be the solution.  
  
  ---
  
  