Answers.Body: <p>Why make this so complicated with the tee command?</p>
  <p>I don't think you even can index an unsorted bam, so you have to wait until sorting is done before you index.  Indexing is really short compared to mapping and sorting, you aren't saving much by trying to do both at once.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T23:17:17.553Z
Answers.LastActivityDate: 2020-10-22T23:17:17.553Z
Answers.UserDisplayName: swbarnes2
Answers.Body: <p>This is an error from <code>bwa</code> saying that it can't find your input files:</p>
  <p>Here's a checklist:</p>
  <ol>
  <li>Check that you have no whitespace after your <code>\</code> newline breaks. I.e. <code>\</code> is the <em>very last character</em></li>
  <li>Check that you have no whitespace in any path names</li>
  <li><em>Quote your variables</em>: <code>$REF_PATH</code> -&gt; <code>&quot;$REF_PATH&quot;</code></li>
  <li>Use <code>set -x</code> at the top of your bash script to debug.</li>
  <li>In general, read <a href="https://kvz.io/bash-best-practices.html" rel="noreferrer">this</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T08:54:09.107Z
Answers.LastActivityDate: 2020-10-22T08:54:09.107Z
Answers.UserDisplayName: Bastian Schiffthaler
Body: <p>I'm trying to use the code specified in <a href="https://biology.stackexchange.com/questions/59493/how-to-convert-bwa-mem-output-to-bam-format-without-saving-sam-file?fbclid=IwAR1oClXdwhtsqvQqkmmc8a1af-dTaALbtkQ2kKW3cGnIrLZCaa_6HxjvrfY">this link</a> to create a .bam file without the need for a .sam file.</p>
  <p>Here is the code I'm using:</p>
  <pre><code>bwa mem -R &quot;@RG\tID:<span class="math-container">$SAMPLE\tPL:illumina\tSM:$</span>SAMPLE&quot; \
      <span class="math-container">$REF_PATH/hg38.fasta \
      mapping/R1.fastq.gz \
      mapping/R2.fastq.gz | \
      samtools sort - | \
      tee mapping/$</span>SAMPLE\.bam | \
      samtools index - mapping/$SAMPLE\.bam.bai \
      2&gt;&gt; log
  </code></pre>
  <p>and I get the following error:</p>
  <pre><code>[M::bwa_idx_load_from_disk] read 0 ALT contigs
  `[E::main_mem] fail to open file ` '.
  </code></pre>
  <p>The ref path and file locations work when creating and converting a .sam to a .bam file.
  Any ideas?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T10:40:29.120Z
Comments.Text: The problem is not in skipping the intermediate `.sam` format, but already the mapping step.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T11:08:45.163Z
Comments.Text: That link does not contain that code — is there a different link it does come from? (Whatever tutorial it's from would benefit from an update. Writing the BAM file via `tee` so that you can pipe to `samtools index` is such a spectacular anti-pattern. Though it's unfortunate that `sort` is one of the subcommands that does not yet support samtools's new `--write-index` option.)
Comments.UserDisplayName: John Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-22T12:12:30.337Z
Comments.Text: Ha, I was wrong, the index is indeed loaded. The reads are not...  The expected log should look like this:
  `mapping
  [M::bwa_idx_load_from_disk] read 0 ALT contigs
  [M::process] read 1006360 sequences (150000291 bp)...
  ...`
  Then I would follow @Bastian Schiffthaler's answer to find what's the problem with your reads.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-19T09:31:05.820Z
Comments.Text: It turns out [`samtools sort` does support `--write-index`](https://github.com/samtools/samtools/pull/1342) since 1.10 after all. So this can be written more simply as `bwa … | samtools sort --write-index -o mapping/$SAMPLE.bam -` (plus a little more if you want to use a BAI index rather than `‑‑write-index`'s default CSI).
Comments.UserDisplayName: John Marshall
Tags: <samtools><bash><bwa>
Title: Trying to create a .bam file without the need for a .sam file
ThreadText: # Trying to create a .bam file without the need for a .sam file
  
  (Asked by: username_0 on 2020-10-22)
  
  I'm trying to use the code specified in [this link](https://biology.stackexchange.com/questions/59493/how-to-convert-bwa-mem-output-to-bam-format-without-saving-sam-file?fbclid=IwAR1oClXdwhtsqvQqkmmc8a1af-dTaALbtkQ2kKW3cGnIrLZCaa_6HxjvrfY) to create a .bam file without the need for a .sam file.
  
  Here is the code I'm using:
  
  ```
  bwa mem -R "@RG\tID:$SAMPLE\tPL:illumina\tSM:$SAMPLE" \
      $REF_PATH/hg38.fasta \
      mapping/R1.fastq.gz \
      mapping/R2.fastq.gz | \
      samtools sort - | \
      tee mapping/$SAMPLE\.bam | \
      samtools index - mapping/$SAMPLE\.bam.bai \
      2>> log
  
  ```
  
  and I get the following error:
  
  ```
  [M::bwa_idx_load_from_disk] read 0 ALT contigs
  `[E::main_mem] fail to open file ` '.
  
  ```
  
  The ref path and file locations work when creating and converting a .sam to a .bam file.
  Any ideas?
  
  **Tags:** samtools, bash, bwa
  
  > **Comments:**
  > **username_1**: The problem is not in skipping the intermediate `.sam` format, but already the mapping step.  
  > **username_2**: That link does not contain that code — is there a different link it does come from? (Whatever tutorial it's from would benefit from an update. Writing the BAM file via `tee` so that you can pipe to `samtools index` is such a spectacular anti-pattern. Though it's unfortunate that `sort` is one of the subcommands that does not yet support samtools's new `--write-index` option.)  
  > **username_1**: Ha, I was wrong, the index is indeed loaded. The reads are not...  The expected log should look like this: `mapping [M::bwa_idx_load_from_disk] read 0 ALT contigs [M::process] read 1006360 sequences (150000291 bp)... ...` Then I would follow @username_4's answer to find what's the problem with your reads.  
  > **username_2**: It turns out [`samtools sort` does support `--write-index`](https://github.com/samtools/samtools/pull/1342) since 1.10 after all. So this can be written more simply as `bwa … | samtools sort --write-index -o mapping/$SAMPLE.bam -` (plus a little more if you want to use a BAI index rather than `‑‑write-index`'s default CSI).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2020-10-22. Score: 5 | Accepted answer
  
  This is an error from `bwa` saying that it can't find your input files:
  
  Here's a checklist:
  
  1. Check that you have no whitespace after your `\` newline breaks. I.e. `\` is the *very last character*
  2. Check that you have no whitespace in any path names
  3. *Quote your variables*: `$REF_PATH` -> `"$REF_PATH"`
  4. Use `set -x` at the top of your bash script to debug.
  5. In general, read [this](https://kvz.io/bash-best-practices.html)
  
  
  ---
  
  ### Answer by username_3 on 2020-10-22. Score: 0
  
  Why make this so complicated with the tee command?
  
  I don't think you even can index an unsorted bam, so you have to wait until sorting is done before you index. Indexing is really short compared to mapping and sorting, you aren't saving much by trying to do both at once.
  
  
  ---
  
  