Answers.Body: <p>You are correct that the concatenation of input read files will confuse BWA. BWA depends on the input reads being in the same order in R1 and R2 files for paired-end alignment. </p>
  
  <p>From <a href="http://bio-bwa.sourceforge.net/bwa.shtml#3" rel="nofollow noreferrer">bwa docs</a>: <code>If mates.fq is present, this command assumes the i-th read in reads.fq and the i-th read in mates.fq constitute a read pair.</code> BWA will not search through FASTQ files trying to find matching mate IDs. </p>
  
  <p>You will notice that your initial error message is reporting two different read IDs that are improperly paired. Presumably the mispaired read corresponds to the first read in the out-of-order file. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T05:24:21.500Z
Answers.LastActivityDate: 2019-11-08T05:24:21.500Z
Answers.UserDisplayName: Maximilian Press
Answers.Body: <p>AS both the OP and @MaximilianPress suggested, <strong>the paired-end reads should be provided as 2 separate arguments to bwa</strong>. This is true whether the reads are in files or streamed. For the sake of completeness, I provide the corrected command here.</p>
  <p><code>bwa mem -M ref_file.fa &lt;(cat A.read1.fq B.read1.fq ...) &lt;(cat A.read2.fq B.read2.fq ...)</code></p>
  <p><strong>The order of the files is important</strong>: if the order is A, B for read1, then it should be A, B and <em>not</em> B, A for read2.</p>
  <h1>EDIT:</h1>
  <p><em>(Thanks to @JohnMarshall for pointing this out.)</em></p>
  <p><strong>Avoid double quotes around process substitution. Quote the file names (if needed).</strong> The quotes around process substitution, as apparently intended, work only on some systems/versions<sup>1</sup>, and seem like an undocumented feature to me.</p>
  <p><strong>DON'T:</strong></p>
  <p><code>bwa mem -M ref_file.fa &quot;&lt;(cat A.read1.fq B.read1.fq ...)&quot; &quot;&lt;(cat A.read2.fq B.read2.fq ...)&quot;</code></p>
  <hr />
  <p><sup>1</sup>
  Oddly enough, process substitution inside double quotes works at least in the case I tested before posting: <code>bwa mem -M test1.fa &quot;&lt;(cat test_assembler_tiny_1.1.fastq)&quot; &quot;&lt;(cat test_assembler_tiny_1.2.fastq)&quot;</code> This contradicts the answer here: <a href="https://unix.stackexchange.com/q/486019">https://unix.stackexchange.com/q/486019</a> . I used bwa 0.7.17-r1188, with both GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17) and zsh 5.7.1 (x86_64-apple-darwin17.7.0), on macOS High Sierra 10.13.6. From OP's error msg, it seems it worked for @glyph too.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-11T09:53:35.803Z
Answers.Text: Quotes around ordinary filenames are optional, but here you've put quotes around the shell redirection metacharacters. This leads to `<(cat …): No such file or directory`. Don't quote the `<()` characters if you want them to be interpreted by the shell.
Answers.UserDisplayName: John Marshall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T14:55:26.733Z
Answers.LastActivityDate: 2019-11-11T19:31:36.617Z
Answers.LastEditDate: 2020-06-18T08:30:41.743Z
Answers.UserDisplayName: Timur Shtatland
Body: <p>Running BWA Mem on a number of paired end fastq files using process substitution on the inputs results in this error:</p>
  
  <pre><code>[mem_sam_pe] paired reads have different names: "HWI-D00360:5:H814YADXX:1:2111:20284:33068", "HWI-D00360:5:H814YADXX:1:1213:17580:34847"
  </code></pre>
  
  <p>The command: </p>
  
  <pre><code>bwa mem -M ref.fa "&lt;(cat U0a_CGATGT_L001_R1_004.fastq U0a_CGATGT_L001_R1_005.fastq U0a_CGATGT_L002_R1_001.fastq U0a_CGATGT_L002_R1_002.fastq U0a_CGATGT_L002_R1_003.fastq U0a_CGATGT_L001_R1_001.fastq U0a_CGATGT_L001_R1_002.fastq U0a_CGATGT_L002_R1_004.fastq U0a_CGATGT_L001_R1_003.fastq U0a_CGATGT_L002_R1_005.fastq)" "&lt;(cat U0a_CGATGT_L001_R2_003.fastq U0a_CGATGT_L002_R2_005.fastq U0a_CGATGT_L001_R2_004.fastq U0a_CGATGT_L001_R2_005.fastq U0a_CGATGT_L002_R2_001.fastq U0a_CGATGT_L002_R2_002.fastq U0a_CGATGT_L001_R2_001.fastq U0a_CGATGT_L002_R2_003.fastq U0a_CGATGT_L001_R2_002.fastq U0a_CGATGT_L002_R2_004.fastq)" &gt; out.sam
  </code></pre>
  
  <p>However the read (there were several) that threw this error does seem to be properly paired: </p>
  
  <pre><code>grep 'HWI-D00360:5:H814YADXX:1:2111:20284:33068' *.fastq
  U0a_CGATGT_L001_R1_004.fastq:@HWI-D00360:5:H814YADXX:1:2111:20284:33068 1:N:0:CGATGT
  U0a_CGATGT_L001_R2_004.fastq:@HWI-D00360:5:H814YADXX:1:2111:20284:33068 2:N:0:CGATGT
  </code></pre>
  
  <p>I can see that the concatenation of the input files is happening in a different order, so this is the next thing I'm going to try to fix, but has anyone else run into this?  </p>
  
  <p>(I probably just answered my own question but I'll post it anyway, that may or may not be the issue) </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T09:59:13.547Z
Comments.Text: In 2019 you almost certainly don't want to be using the `-M` flag. See bwa [#26](https://github.com/lh3/bwa/pull/26), [#96](https://github.com/lh3/bwa/pull/96), [#188](https://github.com/lh3/bwa/issues/188), et al.
Comments.UserDisplayName: John Marshall
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-11T17:25:09.870Z
Comments.Text: Ok thanks @JohnMarshall I'll give it a try without. I probably had it there for the older Picard or GATK but I've since updated both
Comments.UserDisplayName: 111
Tags: <phylogenetics><read-mapping><fastq><bwa><bash>
Title: Error with BWA Mem input having multiple fastq files using cat and process substitution
ThreadText: # Error with BWA Mem input having multiple fastq files using cat and process substitution
  
  (Asked by: username_0 on 2019-11-07)
  
  Running BWA Mem on a number of paired end fastq files using process substitution on the inputs results in this error:
  
  ```
  [mem_sam_pe] paired reads have different names: "HWI-D00360:5:H814YADXX:1:2111:20284:33068", "HWI-D00360:5:H814YADXX:1:1213:17580:34847"
  
  ```
  
  The command:
  
  ```
  bwa mem -M ref.fa "<(cat U0a_CGATGT_L001_R1_004.fastq U0a_CGATGT_L001_R1_005.fastq U0a_CGATGT_L002_R1_001.fastq U0a_CGATGT_L002_R1_002.fastq U0a_CGATGT_L002_R1_003.fastq U0a_CGATGT_L001_R1_001.fastq U0a_CGATGT_L001_R1_002.fastq U0a_CGATGT_L002_R1_004.fastq U0a_CGATGT_L001_R1_003.fastq U0a_CGATGT_L002_R1_005.fastq)" "<(cat U0a_CGATGT_L001_R2_003.fastq U0a_CGATGT_L002_R2_005.fastq U0a_CGATGT_L001_R2_004.fastq U0a_CGATGT_L001_R2_005.fastq U0a_CGATGT_L002_R2_001.fastq U0a_CGATGT_L002_R2_002.fastq U0a_CGATGT_L001_R2_001.fastq U0a_CGATGT_L002_R2_003.fastq U0a_CGATGT_L001_R2_002.fastq U0a_CGATGT_L002_R2_004.fastq)" > out.sam
  
  ```
  
  However the read (there were several) that threw this error does seem to be properly paired:
  
  ```
  grep 'HWI-D00360:5:H814YADXX:1:2111:20284:33068' *.fastq
  U0a_CGATGT_L001_R1_004.fastq:@HWI-D00360:5:H814YADXX:1:2111:20284:33068 1:N:0:CGATGT
  U0a_CGATGT_L001_R2_004.fastq:@HWI-D00360:5:H814YADXX:1:2111:20284:33068 2:N:0:CGATGT
  
  ```
  
  I can see that the concatenation of the input files is happening in a different order, so this is the next thing I'm going to try to fix, but has anyone else run into this?
  
  (I probably just answered my own question but I'll post it anyway, that may or may not be the issue)
  
  **Tags:** phylogenetics, read-mapping, fastq, bwa, bash
  
  > **Comments:**
  > **username_1**: In 2019 you almost certainly don't want to be using the `-M` flag. See bwa [#26](https://github.com/lh3/bwa/pull/26), [#96](https://github.com/lh3/bwa/pull/96), [#188](https://github.com/lh3/bwa/issues/188), et al.  
  > **username_0**: Ok thanks @username_1 I'll give it a try without. I probably had it there for the older Picard or GATK but I've since updated both  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-11-08. Score: 3 | Accepted answer
  
  You are correct that the concatenation of input read files will confuse BWA. BWA depends on the input reads being in the same order in R1 and R2 files for paired-end alignment.
  
  From [bwa docs](http://bio-bwa.sourceforge.net/bwa.shtml#3): `If mates.fq is present, this command assumes the i-th read in reads.fq and the i-th read in mates.fq constitute a read pair.` BWA will not search through FASTQ files trying to find matching mate IDs.
  
  You will notice that your initial error message is reporting two different read IDs that are improperly paired. Presumably the mispaired read corresponds to the first read in the out-of-order file.
  
  
  ---
  
  ### Answer by username_3 on 2019-11-08. Score: 3
  
  AS both the OP and @username_2 suggested, **the paired-end reads should be provided as 2 separate arguments to bwa**. This is true whether the reads are in files or streamed. For the sake of completeness, I provide the corrected command here.
  
  `bwa mem -M ref_file.fa <(cat A.read1.fq B.read1.fq ...) <(cat A.read2.fq B.read2.fq ...)`
  
  **The order of the files is important**: if the order is A, B for read1, then it should be A, B and *not* B, A for read2.
  
  # EDIT:
  
  *(Thanks to @username_1 for pointing this out.)*
  
  **Avoid double quotes around process substitution. Quote the file names (if needed).** The quotes around process substitution, as apparently intended, work only on some systems/versions1, and seem like an undocumented feature to me.
  
  **DON'T:**
  
  `bwa mem -M ref_file.fa "<(cat A.read1.fq B.read1.fq ...)" "<(cat A.read2.fq B.read2.fq ...)"`
  
  ---
  
  1
  Oddly enough, process substitution inside double quotes works at least in the case I tested before posting: `bwa mem -M test1.fa "<(cat test_assembler_tiny_1.1.fastq)" "<(cat test_assembler_tiny_1.2.fastq)"` This contradicts the answer here: <https://unix.stackexchange.com/q/486019> . I used bwa 0.7.17-r1188, with both GNU bash, version 3.2.57(1)-release (x86\_64-apple-darwin17) and zsh 5.7.1 (x86\_64-apple-darwin17.7.0), on macOS High Sierra 10.13.6. From OP's error msg, it seems it worked for @glyph too.
  
  > **Comments:**
  > **username_1**: Quotes around ordinary filenames are optional, but here you've put quotes around the shell redirection metacharacters. This leads to `<(cat …): No such file or directory`. Don't quote the `<()` characters if you want them to be interpreted by the shell.  
  
  ---
  
  