Answers.Body: <p>What does this program do in the expected use case, presumably where intermediate output files are generated? From that, can you get any information about how files are updated? For example, are the creation times on the files identical (as would be expected if they're written at the same time), or are they different from each other?</p>
  <p>I could imagine a situation where the initial cat test works because the two file writes are running concurrently and are non-blocking, whereas the second test doesn't work because it's waiting for input on one of the two files.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-17T22:13:56.290Z
Answers.LastActivityDate: 2024-06-18T13:18:51.890Z
Answers.LastEditDate: 2024-06-18T13:18:51.890Z
Answers.UserDisplayName: gringer
Body: <p>I am trying to use a named pipe to pass data from the Trimmer utility from the <a href="https://www.agilent.com/cs/library/software/Public/AGeNT%20ReadMe.pdf" rel="nofollow noreferrer">AGeNT</a> toolbox from Agilent, through bwa-mem2. The normal behavior is that the Trimmer utility writes trimmed fastq files to disk. I would like to use a named pipe to pass the data to bwa-mem2 for a few reasons:</p>
  <ol>
  <li>speed</li>
  <li>to see if I can.</li>
  </ol>
  <p>I first verified that the the output of Trimmer can in principle be passed through a named pipe. The output of Trimmer was piped to cat whose output was redirected into a text file. The text of this file was compared to a corresponding fastq files that I had previously produced by running Trimmer normally. The text of the files were found to be identical, indicating that output of trimmer was successfully piped.</p>
  <pre><code>#!/usr/bin/bash
  set -e
  set -x
  set -o pipefail
  pipe=True
  ID=345705 
  # untrimmmmed fastq files
  R1=&quot;/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/<span class="math-container">${ID}_S9_R1_001.fastq.gz"
  R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/$</span>{ID}_S9_R2_001.fastq.gz&quot;
  
  # where and how to name the output files
  out_dir=&quot;/mounts/pipes/&quot;
  mkdir -p <span class="math-container">${out_dir}
  out_prefix="$</span>{out_dir}<span class="math-container">${ID}_Cut"
  #for file in /mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/*.fastq.gz; do gunzip -c "$</span>file&quot; &gt; &quot;/path/to/output/directory/<span class="math-container">$(basename "$</span>{file%.gz}&quot;)&quot;; done
  # software paths
  jarfile=&quot;/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar&quot;
  mem2_executable=&quot;/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw&quot;
  ref_genome=&quot;/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta&quot;
  
  # make output pipes
  R1_PIPE=<span class="math-container">${out_prefix}_R1.fastq.gz
  R2_PIPE=$</span>{out_prefix}_R2.fastq.gz
  mkfifo <span class="math-container">${R1_PIPE} $</span>{R2_PIPE}
  # start writer process trimmer in background
  java -Xmx15G -jar <span class="math-container">${jarfile}\
          -fq1 $</span>{R1} \
          -fq2 <span class="math-container">${R2} \
      -v2 \
      -out $</span>{out_prefix} &amp;
  
  trimmerPid=<span class="math-container">$!
  # start writer processes
  cat $</span>{R1_PIPE} &gt; <span class="math-container">${out_dir}$</span>{ID}&quot;R1.txt&quot;&amp;
  cat <span class="math-container">${R2_PIPE} &gt; $</span>{out_dir}${ID}&quot;R2.txt&quot;
  
  wait &quot;<span class="math-container">$trimmerPid"
  # compare the generated text files to the actualtrimmed files calculated on a previous run of trimmer
  # if they agree then the output was successfully piped
  trimdir="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/TRIMMED_FASTQ"
  diff -q "$</span>{trimdir}/<span class="math-container">${ID}_Cut_R1.fastq.gz" $</span>{out_dir}<span class="math-container">${ID}"R1.txt"
  if [ $</span>? -eq 0 ]
  then
      echo &quot;The R1 files are the same.&quot;
  else
      echo &quot;The R1 files are different.&quot;
  fi
  
  diff -q &quot;<span class="math-container">${trimdir}/$</span>{ID}_Cut_R2.fastq.gz&quot; <span class="math-container">${out_dir}$</span>{ID}&quot;R2.txt&quot;
  if [ $? -eq 0 ]
  then
      echo &quot;The R2 files are the same.&quot;
  else
      echo &quot;The R2 files are different.&quot;
  fi
  </code></pre>
  <p>Agilent advised me that in principle it was not possible to catch the output of trimmer in a named pipe. However, given the result of the first experiment in my post. I think their advice was not correct.</p>
  <p>However when I try to pipe instead to bwa-mem2 I have problems. bwa-mem2 starts, loads the reference genome and writes the header of the SAM file, but no reads. Then hangs for eternity.</p>
  <pre><code>#!/usr/bin/bash
  set -e
  set -x
  set -o pipefail
  pipe=True
  ID=345705 
  # untrimmmmed fastq files
  R1=&quot;/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/<span class="math-container">${ID}_S9_R1_001.fastq.gz"
  R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/$</span>{ID}_S9_R2_001.fastq.gz&quot;
  
  # where and how to name the output files
  out_dir=&quot;/mounts/pipes/&quot;
  mkdir -p <span class="math-container">${out_dir}
  out_prefix="$</span>{out_dir}${ID}_Cut&quot;
  
  # software paths
  jarfile=&quot;/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar&quot;
  mem2_executable=&quot;/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw&quot;
  ref_genome=&quot;/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta&quot;
  
  # make output pipes
  R1_PIPE=<span class="math-container">${out_prefix}_R1.fastq.gz
  R2_PIPE=$</span>{out_prefix}_R2.fastq.gz
  mkfifo <span class="math-container">${R1_PIPE} $</span>{R2_PIPE}
  #cat &lt;(zcat sample_1.fq.gz | head -n 4) &lt;(zcat sample_2.fq.gz | head -n 4) | bwa-mem2 mem -t 4 -p ref.fa - &gt; out.mem2.sam
  # start writer process trimmer in background
  java -Xmx15G -jar <span class="math-container">${jarfile}\
      -fq1 $</span>{R1} \
          -fq2 <span class="math-container">${R2} \
      -v2 \
      -out $</span>{out_prefix} &amp;
  
  trimmerPid=<span class="math-container">$!
  $</span>{mem2_executable} mem -t 7 -C -R &quot;@RG\tID:<span class="math-container">${ID}\tPL:ILLUMINA\tLB:SureSelectXTHSv2\tSM:$</span>{ID}&quot;\
           <span class="math-container">${ref_genome} $</span>{R1_PIPE} <span class="math-container">${R2_PIPE} &gt; $</span>{out_dir}${ID}.sam
  
  wait &quot;$trimmerPid&quot;
  </code></pre>
  <p>Any help you can give is much appreciated.</p>
  <p><strong>Update</strong></p>
  <p>It looks like the problem may be that bwa-mem2 cannot read from the named pipe. The output of strace indicates a failed call to lseek immediately after opening the the pipes.</p>
  <p>openat(AT_FDCWD, &quot;/mounts/pipes/345705_Cut_R1.fastq.gz&quot;, O_RDONLY) = 4</p>
  <p>lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)</p>
  
Tags: <phylogenetics><linux><shell>
Title: Passing data from the Agilent Trimmer utility to bwa-mem2 via a named pipe
ThreadText: # Passing data from the Agilent Trimmer utility to bwa-mem2 via a named pipe
  
  (Asked by: username_0 on 2024-06-17)
  
  I am trying to use a named pipe to pass data from the Trimmer utility from the [AGeNT](https://www.agilent.com/cs/library/software/Public/AGeNT%20ReadMe.pdf) toolbox from Agilent, through bwa-mem2. The normal behavior is that the Trimmer utility writes trimmed fastq files to disk. I would like to use a named pipe to pass the data to bwa-mem2 for a few reasons:
  
  1. speed
  2. to see if I can.
  
  I first verified that the the output of Trimmer can in principle be passed through a named pipe. The output of Trimmer was piped to cat whose output was redirected into a text file. The text of this file was compared to a corresponding fastq files that I had previously produced by running Trimmer normally. The text of the files were found to be identical, indicating that output of trimmer was successfully piped.
  
  ```
  #!/usr/bin/bash
  set -e
  set -x
  set -o pipefail
  pipe=True
  ID=345705 
  # untrimmmmed fastq files
  R1="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R1_001.fastq.gz"
  R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R2_001.fastq.gz"
  
  # where and how to name the output files
  out_dir="/mounts/pipes/"
  mkdir -p ${out_dir}
  out_prefix="${out_dir}${ID}_Cut"
  #for file in /mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/*.fastq.gz; do gunzip -c "$file" > "/path/to/output/directory/$(basename "${file%.gz}")"; done
  # software paths
  jarfile="/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar"
  mem2_executable="/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw"
  ref_genome="/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta"
  
  # make output pipes
  R1_PIPE=${out_prefix}_R1.fastq.gz
  R2_PIPE=${out_prefix}_R2.fastq.gz
  mkfifo ${R1_PIPE} ${R2_PIPE}
  # start writer process trimmer in background
  java -Xmx15G -jar ${jarfile}\
          -fq1 ${R1} \
          -fq2 ${R2} \
      -v2 \
      -out ${out_prefix} &
  
  trimmerPid=$!
  # start writer processes
  cat ${R1_PIPE} > ${out_dir}${ID}"R1.txt"&
  cat ${R2_PIPE} > ${out_dir}${ID}"R2.txt"
  
  wait "$trimmerPid"
  # compare the generated text files to the actualtrimmed files calculated on a previous run of trimmer
  # if they agree then the output was successfully piped
  trimdir="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/TRIMMED_FASTQ"
  diff -q "${trimdir}/${ID}_Cut_R1.fastq.gz" ${out_dir}${ID}"R1.txt"
  if [ $? -eq 0 ]
  then
      echo "The R1 files are the same."
  else
      echo "The R1 files are different."
  fi
  
  diff -q "${trimdir}/${ID}_Cut_R2.fastq.gz" ${out_dir}${ID}"R2.txt"
  if [ $? -eq 0 ]
  then
      echo "The R2 files are the same."
  else
      echo "The R2 files are different."
  fi
  
  ```
  
  Agilent advised me that in principle it was not possible to catch the output of trimmer in a named pipe. However, given the result of the first experiment in my post. I think their advice was not correct.
  
  However when I try to pipe instead to bwa-mem2 I have problems. bwa-mem2 starts, loads the reference genome and writes the header of the SAM file, but no reads. Then hangs for eternity.
  
  ```
  #!/usr/bin/bash
  set -e
  set -x
  set -o pipefail
  pipe=True
  ID=345705 
  # untrimmmmed fastq files
  R1="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R1_001.fastq.gz"
  R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R2_001.fastq.gz"
  
  # where and how to name the output files
  out_dir="/mounts/pipes/"
  mkdir -p ${out_dir}
  out_prefix="${out_dir}${ID}_Cut"
  
  # software paths
  jarfile="/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar"
  mem2_executable="/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw"
  ref_genome="/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta"
  
  # make output pipes
  R1_PIPE=${out_prefix}_R1.fastq.gz
  R2_PIPE=${out_prefix}_R2.fastq.gz
  mkfifo ${R1_PIPE} ${R2_PIPE}
  #cat <(zcat sample_1.fq.gz | head -n 4) <(zcat sample_2.fq.gz | head -n 4) | bwa-mem2 mem -t 4 -p ref.fa - > out.mem2.sam
  # start writer process trimmer in background
  java -Xmx15G -jar ${jarfile}\
      -fq1 ${R1} \
          -fq2 ${R2} \
      -v2 \
      -out ${out_prefix} &
  
  trimmerPid=$!
  ${mem2_executable} mem -t 7 -C -R "@RG\tID:${ID}\tPL:ILLUMINA\tLB:SureSelectXTHSv2\tSM:${ID}"\
           ${ref_genome} ${R1_PIPE} ${R2_PIPE} > ${out_dir}${ID}.sam
  
  wait "$trimmerPid"
  
  ```
  
  Any help you can give is much appreciated.
  
  **Update**
  
  It looks like the problem may be that bwa-mem2 cannot read from the named pipe. The output of strace indicates a failed call to lseek immediately after opening the the pipes.
  
  openat(AT\_FDCWD, "/mounts/pipes/345705\_Cut\_R1.fastq.gz", O\_RDONLY) = 4
  
  lseek(4, 0, SEEK\_CUR) = -1 ESPIPE (Illegal seek)
  
  **Tags:** phylogenetics, linux, shell
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-06-17. Score: 2
  
  What does this program do in the expected use case, presumably where intermediate output files are generated? From that, can you get any information about how files are updated? For example, are the creation times on the files identical (as would be expected if they're written at the same time), or are they different from each other?
  
  I could imagine a situation where the initial cat test works because the two file writes are running concurrently and are non-blocking, whereas the second test doesn't work because it's waiting for input on one of the two files.
  
  
  ---
  
  